Thursday, August 8, 2013

From Raspberry Pi to Azure Service Bus using AMQP 1.0 (Proton-C)

Thursday, August 08, 2013 Posted by Andre Broers , , , , , , , , , , 8 comments
To measure my energymeter I needed a low power solution to post the measurement data to the cloud.

I have a sensor which generates time ticks which I want to upload to the an Azure Service Bus. The ticks are read from a gpio port on my raspberry pi. This whole process is running in a c program to efficiently read the interupts from the gpio port in user space. (I will explain this proces in a later post).
Because this process is running as a c program, it makes sense to do the upload to Azure Service Bus also in c. I choose to use the apache Qpid Proton-C library. (qpid-proton). This is c library which speaks AMQP 1.0 and even in the SSL flavour which is needed for Azure Service Bus.

Installing the library on Raspberry Pi

We to install some dependent packages:

sudo apt-get install cmake uuid-dev libssl-dev

Download and unpack the package:

pi@raspberrypi ~ $ wget http://mirrors.supportex.net/apache/qpid/proton/0.4/qpid-proton-0.4.tar.gz
pi@raspberrypi ~ $ tar zxvf qpid-proton-0.4.tar.gz
pi@raspberrypi ~ $ cd qpid-proton-0.4

For some reason the cmake script doesn't find the uuid library. So the change below helps it to choose the right setting.

pi@raspberrypi ~/qpid-proton-0.4 $ cd proton-c
pi@raspberrypi ~/qpid-proton-0.4/proton-c $ vi CMakeLists.txt

replace: 

CHECK_SYMBOL_EXISTS(uuid_generate "uuid/uuid.h" UUID_GENERATE_IN_LIBC)
if (UUID_GENERATE_IN_LIBC)
  list(APPEND PLATFORM_DEFINITIONS "USE_UUID_GENERATE")
else (UUID_GENERATE_IN_LIBC)
  CHECK_LIBRARY_EXISTS (uuid uuid_generate "" UUID_GENERATE_IN_UUID)
  if (UUID_GENERATE_IN_UUID)
    set (UUID_LIB uuid)
    list(APPEND PLATFORM_DEFINITIONS "USE_UUID_GENERATE")
  else (UUID_GENERATE_IN_UUID)
    CHECK_SYMBOL_EXISTS(uuid_create "uuid.h" UUID_CREATE_IN_LIBC)
    if (UUID_CREATE_IN_LIBC)
      list(APPEND PLATFORM_DEFINITIONS "USE_UUID_CREATE")
    else (UUID_CREATE_IN_LIBC)
      CHECK_SYMBOL_EXISTS(UuidToString "rpc.h" WIN_UUID)
      if (WIN_UUID)
        list(APPEND PLATFORM_DEFINITIONS "USE_WIN_UUID")
      else (WIN_UUID)
        message(FATAL_ERROR "No Uuid API found")
      endif (WIN_UUID)
    endif (UUID_CREATE_IN_LIBC)
  endif (UUID_GENERATE_IN_UUID)
endif (UUID_GENERATE_IN_LIBC)

with the lines:

    set (UUID_LIB uuid)
    list(APPEND PLATFORM_DEFINITIONS "USE_UUID_GENERATE")

save the file.

pi@raspberrypi ~/qpid-proton-0.4/proton-c $ cd ..
pi@raspberrypi ~/qpid-proton-0.4 $ mkdir build
pi@raspberrypi ~/qpid-proton-0.4 $ cd build
pi@raspberrypi ~/qpid-proton-0.4/build $ cmake ..
-- The C compiler identification is GNU 4.6.3
-- Check for working C compiler: /usr/bin/gcc
-- Check for working C compiler: /usr/bin/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
PN_VERSION: 0.4
-- Could NOT find Java (missing:  Java_JAVA_EXECUTABLE Java_JAR_EXECUTABLE Java_JAVAC_EXECUTABLE Java_JAVAH_EXECUTABLE Java_JAVADOC_EXECUTABLE)
-- Found OpenSSL: /usr/lib/arm-linux-gnueabihf/libssl.so;/usr/lib/arm-linux-gnueabihf/libcrypto.so (found version "1.0.1e")
-- Looking for clock_gettime
-- Looking for clock_gettime - not found.
-- Looking for clock_gettime in rt
-- Looking for clock_gettime in rt - found
-- Looking for strerror_r
-- Looking for strerror_r - found
-- Looking for atoll
-- Looking for atoll - found
-- Could NOT find SWIG (missing:  SWIG_EXECUTABLE SWIG_DIR)
-- Could NOT find Doxygen (missing:  DOXYGEN_EXECUTABLE)
-- Cannot find both Java and Maven: testing disabled for Proton-J and JNI Bindings
-- Configuring done
-- Generating done
-- Build files have been written to: /home/pi/qpid-proton-0.4/build
pi@raspberrypi ~/qpid-proton-0.4/build $ make all
[  4%] Generating protocol.h
[  9%] Generating encodings.h
Scanning dependencies of target qpid-proton
[ 13%] Building C object proton-c/CMakeFiles/qpid-proton.dir/src/util.c.o
[ 18%] Building C object proton-c/CMakeFiles/qpid-proton.dir/src/error.c.o
[ 22%] Building C object proton-c/CMakeFiles/qpid-proton.dir/src/buffer.c.o
[ 27%] Building C object proton-c/CMakeFiles/qpid-proton.dir/src/parser.c.o
[ 31%] Building C object proton-c/CMakeFiles/qpid-proton.dir/src/scanner.c.o
[ 36%] Building C object proton-c/CMakeFiles/qpid-proton.dir/src/types.c.o
[ 40%] Building C object proton-c/CMakeFiles/qpid-proton.dir/src/framing/framing.c.o
[ 45%] Building C object proton-c/CMakeFiles/qpid-proton.dir/src/codec/codec.c.o
[ 50%] Building C object proton-c/CMakeFiles/qpid-proton.dir/src/dispatcher/dispatcher.c.o
[ 54%] Building C object proton-c/CMakeFiles/qpid-proton.dir/src/engine/engine.c.o
[ 59%] Building C object proton-c/CMakeFiles/qpid-proton.dir/src/message/message.c.o
[ 63%] Building C object proton-c/CMakeFiles/qpid-proton.dir/src/sasl/sasl.c.o
[ 68%] Building C object proton-c/CMakeFiles/qpid-proton.dir/src/messenger.c.o
[ 72%] Building C object proton-c/CMakeFiles/qpid-proton.dir/src/posix/driver.c.o
[ 77%] Building C object proton-c/CMakeFiles/qpid-proton.dir/src/platform.c.o
[ 81%] Building C object proton-c/CMakeFiles/qpid-proton.dir/src/ssl/openssl.c.o
Linking C shared library libqpid-proton.so
[ 81%] Built target qpid-proton
Scanning dependencies of target proton
[ 86%] Building C object proton-c/CMakeFiles/proton.dir/src/proton.c.o
Linking C executable proton
[ 86%] Built target proton
Scanning dependencies of target proton-dump
[ 90%] Building C object proton-c/CMakeFiles/proton-dump.dir/src/proton-dump.c.o
Linking C executable proton-dump
[ 90%] Built target proton-dump
Scanning dependencies of target recv
[ 95%] Building C object proton-c/examples/messenger/c/CMakeFiles/recv.dir/recv.c.o
Linking C executable recv
[ 95%] Built target recv
Scanning dependencies of target send
[100%] Building C object proton-c/examples/messenger/c/CMakeFiles/send.dir/send.c.o
Linking C executable send
[100%] Built target send
pi@raspberrypi ~/qpid-proton-0.4/build $ mkdir ~/lib
pi@raspberrypi ~/qpid-proton-0.4/build $ cd proton-c
pi@raspberrypi ~/qpid-proton-0.4/build/proton-c $ cp libqpid-proton.so.* ~/lib
pi@raspberrypi ~/qpid-proton-0.4/build/proton-c $ cd ../../proton-c/include
pi@raspberrypi ~/qpid-proton-0.4/proton-c/include $ mkdir ~/include
pi@raspberrypi ~/qpid-proton-0.4/proton-c/include $ cp -r proton ~/include

Now we have everything in place to use the library. There are two examples in the tar distribution. The send has a flaw. It doesn't sent the right message. I will create the send and recv examples myself.

pi@raspberrypi ~/qpid-proton-0.4/proton-c/include $ cd ~
pi@raspberrypi ~ $ vi send.c

paste the following code:

#include "proton/message.h"
#include "proton/messenger.h"

#if defined(_WIN32) && ! defined(__CYGWIN__)
#include "../../../proton-c/wincompat/getopt.h"
#else
#include <getopt.h>
#endif
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <ctype.h>

#define check(messenger)                                         \
  {                                                              \
    if(pn_messenger_errno(messenger))                            \
    {                                                            \
      die(__FILE__, __LINE__, pn_messenger_error(messenger));    \
    }                                                            \
  }                                                              \

void die(const char *file, int line, const char *message)
{
  fprintf(stderr, "%s:%i: %s\n", file, line, message);
  exit(1);
}

void usage()
{
  printf("Usage: send [-a addr] [message]\n");
  printf("-a     \tThe target address [amqp[s]://domain[/name]]\n");
  printf("message\tA text string to send.\n");
  exit(0);
}

int main(int argc, char** argv)
{
  int c;
  opterr = 0;
  char * address = (char *) "amqp://0.0.0.0";
  char * msgtext = (char *) "Hello World!";

  while((c = getopt(argc, argv, "ha:b:c:")) != -1)
  {
    switch(c)
    {
    case 'a': address = optarg; break;
    case 'h': usage(); break;

    case '?':
      if(optopt == 'a')
      {
        fprintf(stderr, "Option -%c requires an argument.\n", optopt);
      }
      else if(isprint(optopt))
      {
        fprintf(stderr, "Unknown option `-%c'.\n", optopt);
      }
      else
      {
        fprintf(stderr, "Unknown option character `\\x%x'.\n", optopt);
      }
      return 1;
    default:
      abort();
    }
  }

  if((argc - optind) > 0) msgtext = argv[argc - optind + 2];

  printf("%s",msgtext);

  pn_message_t * message;
  pn_messenger_t * messenger;

  message = pn_message();
  messenger = pn_messenger(NULL);

  pn_messenger_start(messenger);

  pn_message_set_address(message, address);
  pn_data_t *body = pn_message_body(message);
  pn_data_put_string(body, pn_bytes(strlen(msgtext), msgtext));
  pn_messenger_put(messenger, message);
  check(messenger);
  pn_messenger_send(messenger);
  check(messenger);

  pn_messenger_stop(messenger);
  pn_messenger_free(messenger);
  pn_message_free(message);

  return 0;
}

pi@raspberrypi ~ $ vi recv.c

paste the following code:

#include "proton/message.h"
#include "proton/messenger.h"

#if defined(_WIN32) && ! defined(__CYGWIN__)
#include "../../../proton-c/wincompat/getopt.h"
#else
#include <getopt.h>
#endif
#include <stdio.h>
#include <stdlib.h>
#include <ctype.h>

#define check(messenger)                                         \
  {                                                              \
    if(pn_messenger_errno(messenger))                            \
    {                                                            \
      die(__FILE__, __LINE__, pn_messenger_error(messenger));    \
    }                                                            \
  }                                                              \

void die(const char *file, int line, const char *message)
{
  fprintf(stderr, "%s:%i: %s\n", file, line, message);
  exit(1);
}

void usage()
{
  printf("Usage: recv [options] <addr>\n");
  printf("-c    \tPath to the certificate file.\n");
  printf("-k    \tPath to the private key file.\n");
  printf("-p    \tPassword for the private key.\n");
  printf("<addr>\tAn address.\n");
  exit(0);
}

int main(int argc, char** argv)
{
  char* certificate = NULL;
  char* privatekey = NULL;
  char* password = NULL;
  char* address = (char *) "amqp://~0.0.0.0";
  int c;
  opterr = 0;

  while((c = getopt(argc, argv, "hc:k:p:")) != -1)
  {
    switch(c)
    {
    case 'h':
      usage();
      break;

    case 'c': certificate = optarg; break;
    case 'k': privatekey = optarg; break;
    case 'p': password = optarg; break;

    case '?':
      if(optopt == 'c' ||
         optopt == 'k' ||
         optopt == 'p')
      {
        fprintf(stderr, "Option -%c requires an argument.\n", optopt);
      }
      else if(isprint(optopt))
      {
        fprintf(stderr, "Unknown option `-%c'.\n", optopt);
      }
      else
      {
        fprintf(stderr, "Unknown option character `\\x%x'.\n", optopt);
      }
      return 1;
    default:
      abort();
    }
  }

  if((argc - optind) > 0)
  {
    address = argv[argc - optind];
  }

  pn_message_t * message;
  pn_messenger_t * messenger;

  message = pn_message();
  messenger = pn_messenger(NULL);


  if(certificate)
  {
    pn_messenger_set_certificate(messenger, certificate);
  }

  if(privatekey)
  {
    pn_messenger_set_private_key(messenger, privatekey);
  }

  if(password)
  {
    pn_messenger_set_password(messenger, password);
  }

  pn_messenger_start(messenger);
  check(messenger);

  pn_messenger_subscribe(messenger, address);
  check(messenger);

  for(;;)
  {
    pn_messenger_recv(messenger, 1024);
    check(messenger);

    while(pn_messenger_incoming(messenger))
    {
      pn_messenger_get(messenger, message);
      check(messenger);

      char buffer[1024];
      size_t buffsize = sizeof(buffer);
      pn_data_t *body = pn_message_body(message);
      pn_data_format(body, buffer, &buffsize);

      printf("Address: %s\n", pn_message_get_address(message));
      const char* subject = pn_message_get_subject(message);
      printf("Subject: %s\n", subject ? subject : "(no subject)");
      printf("Content: %s\n", buffer);
    }
  }

  return 0;
}

Now it's time to build the code using our new library:

gcc -I/home/pi/include -L/home/pi/lib -lqpid-proton send.c -o send
gcc -I/home/pi/include -L/home/pi/lib -lqpid-proton recv.c -o recv

Now we have the two executables to send and receive a message.

Now it is time to create the Azure Service Bus namespace and a queue and a topic with a subscriber.

Go to the Windows Azure Management portal and go to the tab Service Bus. Press Create to create a new namespace. Give it a name.




After this we have a namespace. Now it is time to find the owner credentials of this namespace. We need to remember these because later on we have to login using these credentials.
Select the new namespace and click on Connection Information. Remember the username and the password.

Now it is time to create a queue. We name it: queue. Open the namespace. Click on the tab Queue and select Create. Create a queue named queue and leave the rest default.



For this example we also create a Topic named: topic and a subscriber to this topic named: subscriber1. Leave the rest of the option default.





Try our send and recv programs 

Now it is time to send a message from our c program on the raspberry pi.
First thing to do is set the library path so that the dynamic linker can find our amqp library:

export LD_LIBRARY_PATH=/home/pi/lib

Now fill the queue:

pi@raspberrypi ~ $ ./send -a amqps://owner:<long password>@bekijkhet.servicebus.windows.net/queue QueueMessage123
pi@raspberrypi ~ $ ./send -a amqps://owner:<long password>@bekijkhet.servicebus.windows.net/queue QueueMessage456
pi@raspberrypi ~ $

We send two messages to the queue Queue:
Now it's time to read them back. Usually not on the raspberry pi ;-) but for this demo it makes the round trip.

pi@raspberrypi ~ $ ./recv amqps://owner:<long password>@bekijkhet.servicebus.windows.net/queue
Address: amqps://owner:<long password>@bekijkhet.servicebus.windows.net/queue
Subject: (no subject)
Content: "QueueMessage123"
Address: amqps://owner:<long password>@bekijkhet.servicebus.windows.net/queue
Subject: (no subject)
Content: "QueueMessage456"
^C
pi@raspberrypi ~ $

It is showing the messages in the order we put them in the queue.

Now a test with the topic and the subscription:

pi@raspberrypi ~ $ ./send -a amqps://owner:<long password>@bekijkhet.servicebus.windows.net/topic TopicMessage123
pi@raspberrypi ~ $ ./send -a amqps://owner:<long password>@bekijkhet.servicebus.windows.net/topic TopicMessage456
pi@raspberrypi ~ $

We can see it has forwarded the messages to our subscription:

Now read the messages from our subscription:

pi@raspberrypi ~ $ ./recv amqps://owner:<longpassword>@bekijkhet.servicebus.windows.net/topic/Subscriptions/subscription1
Address: amqps://owner:<longpassword>@bekijkhet.servicebus.windows.net/topic
Subject: (no subject)
Content: "TopicMessage123"
Address: amqps://owner:<longpassword>@bekijkhet.servicebus.windows.net/topic
Subject: (no subject)
Content: "TopicMessage456"
^C
pi@raspberrypi ~ $

This makes it possible to publish data from small hardware to the cloud where the processing power is.
One step closer to the Internet Of Things (IoT)

8 comments:

  1. Hi,

    Thanks for this guide. I am new to Linux and GCC and I got a bit stuck while following your instructions. I can't get the two programs to build. This is the error I get when executing gcc:

    pi@raspberrypi ~ $ gcc -I/home/pi/include -L/home/pi/lib -l/qpid-proton send.c -o send
    /usr/bin/ld: cannot find -l/qpid-proton
    collect2: ld returned 1 exit status

    Do you have any advice?

    Thanks ahead!

    ReplyDelete
  2. Hi Anonymous,

    Did you find an answer?

    I ran into this problem doing this on a ubuntu server very frustrating.

    I found, moving the library files to /usr/lib/ instead of ~/lib worked but I am having trouble connecting to a service bus :(

    Is there any test service busses running that us n00bs can use?

    Thanks

    ReplyDelete