Bringing MQTT authentication and REST together

For most non-trivial server software which run in production, authentication is a very important. Of course this also applies to MQTT brokers. Luckily the MQTT v3.1 specification includes a username and password authentication mechanism which most MQTT brokers implement.

While username/password files for authentication are sufficient for playing around with MQTT brokers, they are not sufficient for enterprise-grade production systems. When integrating a MQTT broker in existing software landscapes, typically there are existing databases and services.

To demonstrate how dead simple it is to integrate an existing HTTP REST API for MQTT authentication, I created a simple HiveMQ MQTT broker plugin which delegates the authentication mechanism to the REST API (in this case a mock REST API). This API returns a JSON response which we parse in the HiveMQ plugin. It uses the excellent Apache HTTPClient from the HTTP Components project to integrate the authentication mechanism. You can find the project on Github here: https://github.com/dobermai/hivemq-rest-auth-plugin.

HiveMQ + REST API

This shows the whole implementation of the authentication mechanism. Feels like 90% exception handling ;-)

The exact same mechanism can be used if you want to integrate MQTT authentication with some SOAP webservices, NoSQL databases, SQL databases, OAuth and anything you can imagine.

Make sure your MQTT broker of choice supports plugins. If you want to give the HiveMQ plugin system a shot, start here.

Installing a HiveMQ MQTT Server on AWS EC2 with enabled Websockets

To enable communication between MQTT devices, it’s necessary to use a MQTT broker as the central server for your M2M communication. Although there are some public brokers available like mqtt-dashboard.com, it’s a good idea to set up an own server for playing around. This post shows how to set up a HiveMQ MQTT server instance on Amazon Web Services Elastic Compute Cloud (EC2). As an additional goodie, we want to enable MQTT over websockets, so every browser can be a full-featured MQTT client!

Step 1: Create a new EC2 instance

General

The first step is to launch a new EC2 instance. In general it does not matter which OS you choose for HiveMQ as it runs perfectly on every major OS. Any Linux distribution should be fine, I will use a Ubuntu 12.04 LTS. To get started, a Micro Instance will be sufficient, if you need real power and throughput, you should start with more RAM and more vCPUs.

Security groups

Security Groups are very important to configure correctly, otherwise we won’t be able to connect to our server correctly.

AWS MQTT Security Group Settings

AWS MQTT Security Group Settings

Open the following ports to the outside world for maximum MQTT pleasure:

  • 22: Needed for SSH. You probably lock yourself out if your don’t have this port open. Consider restricting this port to your IP adress(es) only.
  • 1883: The MQTT standard port
  • 8883: The MQTT standard port for MQTT over TLS.
  • 8000: The port we want to use for MQTT over websockets

Step 2: Download and install HiveMQ

After launching EC2 instance, we should SSH into it to install Java and HiveMQ. Depending on your OS, these commands might be a bit different.

Install Java + Utils

First we want to install Java and needed utilities. Execute the following commands:

sudo apt-get update
sudo apt-get install openjdk-7-jre-headless unzip

Now you can run

java -version

and the output should look like this:

java version "1.7.0_25"
OpenJDK Runtime Environment (IcedTea 2.3.10) (7u25-2.3.10-1ubuntu0.12.04.2)
OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)

Install HiveMQ

Now we just have do download HiveMQ and unzip it. Execute the following:

wget --content-disposition http://www.hivemq.com/downloads/releases/latest
unzip hivemq-1.x.x
cd hivemq-1.x.x

Configure HiveMQ

HiveMQ comes with sensible defaults and we could get started without modifying the configuration if we don’t need websockets support. But since MQTT over websockets is just awesome, we’ll enable it:

Edit the conf/configuration.properties file and change the following values:

websockets.enabled=true
websockets.port=8000

Step 3: Start HiveMQ

Now just run bin/run.sh and HiveMQ should start up. Verify to see an output like this:

2013-11-30 23:04:46,872 INFO  - HiveMQ home directory: /home/ubuntu/hivemq-1.4.2
2013-11-30 23:04:46,876 INFO  - Starting HiveMQ Server
2013-11-30 23:04:50,225 WARN  - No license file found. Using free personal licensing with restrictions to 25 connections.
2013-11-30 23:04:50,832 INFO  - Activating statistics callbacks with an interval of 60 seconds
2013-11-30 23:04:50,833 INFO  - Activating $SYS topics with an interval of 60 seconds
2013-11-30 23:04:52,053 INFO  - Starting on all interfaces and port 1883
2013-11-30 23:04:52,069 INFO  - Starting with Websockets support on all interfaces and port 8000
2013-11-30 23:04:52,076 INFO  - Started HiveMQ 1.4.2 in 5207ms

That was all. Now you have a high performance MQTT server up and running in the cloud and you can start writing your MQTT applications on devices AND in the browser.

P.S. You can test the MQTT over websockets support with the nifty Websocket Browser Client here

EclipseCon 2013

I’m back from the EclipseCon Europe in Ludwigsburg and it was a great conference, I met many new and old friends and had lots of fun. The conference was very well organized and as far as I heard it was the biggest EclipseCon Europe ever.

Although I didn’t visit all the sessions I wanted to attend (to many good conversations in the hallways!), I at least managed to see the keynotes. :-) The keynote “Single Points of Failure: The Human Element of Software Engineering” by Brian Fitzpatrick was brilliant. Very entertaining and non-technical keynote.

The second keynote, “Scaling Pinterest” by Marty Weiner was very insightful and I was amazed and shocked how fast some startups like Pinterest grow and which problems they have to solve. I think Weiner had every NoSQL database you can think of on his slides.

When visiting the conference, it was clear to everyone, that M2M was one of the most important topics this year. In fact, most people were surprised how fast the M2M ecosystem of Eclipse has grown the last two years. Many people (jokingly?) speculated when the number of M2M projects will surpass the “classic” Eclipse projects.

I talked with many people about MQTT, Eclipse Paho and HiveMQ and it was great to see that most people didn’t know much about MQTT before and most of them got very excited about this neat little technology. I personally think the conference was a big success for Eclipse Paho, there was a workshop (Christian Götz and me) about Java Paho and Javascript Paho, there was a regular Session about Paho and many people were also talking about it on the hallways.

Also, the other M2M projects on Eclipse were parts of many discussions on the hallways. I personally think there is some confusion about the focus of all these projects, but luckily Benjamin Cabé did a session about all these M2M projects and hopefully cleared the confusion for some people.

Although there are many people aware of the M2M projects in the Eclipse universe and these projects gained attention and traction recently, I think we have a long way to go until the M2M projects of Eclipse get “mainstream enough” to be an integral part of every (Java) developers toolbox. But if Eclipse continues to follow the path they chose with M2M, I’m confident there is an even brighter future for the M2M projects and Eclipse as Foundation in General.

The slides for my sessions are available here:

Bringing M2M to the web with Paho – Connecting Java devices and online Dashboards with MQTT M2M for Java developers – MQTT with Eclipse Paho

Review of “Getting started with Google Guava”

I recently read the book Getting Started with Google Guava by Bill Bejeck, published by PACKT Publishing. I have been working with Google Guava a long time and I was surprised how many new things I learned from the book, because I thought I knew this library pretty well.

The book itself is very practical and there are tons of code examples. I especially liked the fact that most code examples are written as JUnit tests, so you immediately get a feel of how to use the utility class under discussion and in which context they can be used. I want to discuss each chapter in detail now:

Basic Guava Utilities

This chapter discusses utils from Guava which you will find in many projects. The author explains the advantages of the Guava classes for String manipulations like Splitter, Joiner, Strings and Charmatcher. This chapter is pretty straightforward and if you don’t know these utilities yet, you’ll immediately see how these utils can simplify your life. Guavas Preconditions class and different builders for toString(), hashCode() and equals() methods are also discussed in this chapter.

Functional Programming

One of the most amazing features of the Guava library are the utilities to enable functional-style programming in Java. The author discusses in-depth when to use this functional paradigm and when it’s better to stick with the imperative classic Java approach. You’ll find many examples for the use of the Function and Predicate classes of Guava. Although Suppliers are introduced in this chapter, too, I personally think the Supplier classes were not discussed extensive enough because in my opinion Guavas Suppliers are way too underrated and I use them very frequently.

Collections

The Guava originated from the google-collections library and so it’s no surprise that a big part of Guava are utils for working with Collections or completely new collection classes. The book demonstrates the use of all relevant collection utils and covers additional collection classes like Immutable Collections, Table, BiMap, etc. This chapter also introduces the Ordering class as a complement to classic Comparators.

Concurrency

Although very short, this is one of the chapters I enjoyed most in the book. It covers the most important classes in Guava for dealing with concurrency and introduces Monitor, ListenableFuture, FutureCallback, AsyncFunction and RateLimiter. I learned a lot here and especially Monitor and ListenableFuture are classes which I intend to use more. Especially ListenableFuture is interesting as it can completely replace classic Java Futures and are more powerful.

Cache

Guavas Cache implementation is very useful and is a lightweight alternative to libraries like EHCache. It’s widely adopted and so it’s no surprise that the Cache gets an extra chapter. For people who are looking for a lightweight, configurable and powerful local cache implementation, this chapter covers everything you will ever need to know about Guavas Cache.

Eventbus

I was excited to see that Guavas Eventbus is covered in a whole chapter because I use the EventBus regularly and it’s an easy and lightweight way to decouple your application. The chapter discusses how to use the event bus and and what advantage you get in your application using it. The code examples are great and show how easy it is to decouple your application components. Finally it’s discussed how to integrate the Eventbus with your Dependency Injection Framework for convenient usage. The code examples are for spring which is a bit irritating, I personally think Google Guice would be a better fit. The concepts for integrating are the same, though.

Files

The eight chapter discusses the shortcomings of classic Java File handling and how Guava fills this hole. It introduces many utility classes for manipulating files, file contents and folders. I personally get bored quickly when reading about file handling, but this is nothing I can blame the author for.

Other utilities

The last chapter covers some other utilities form Guava which are worth mentioning. It introduces Hashing functionality, Bloom Filters, and the super handy Optional class. I am using the later heavily and I think this class should have deserved more than just a very short introduction.

Conclusion

I was very excited from the book, it covers all relevant utils from Guava thoroughly and I learned a lot. The author decided to let code speak and included many code snippets for every topic discussed in the book. I can absolutely recommend this practical book for anyone who is not familiar with Guava (yet) and even long time users of the library will learn some new and exciting stuff.

MQTT Table Football with Arduino, Raspberry Pi and Websockets

One of our main motivation refreshers in our office is our football table and we use it heavily every day. Someone came up with the idea: “Hey, why don’t we add some freaking MQTT support to the football table?”. Of course there was no argument against it and so we added the MQTT support and used a Arduino One for the job. To make things more interesting we decided against a mechanical goal trigger and used infrared sensors for detecting goals. To raise the motivation even more we used a Raspberry Pi to play goal celebration sounds on actual goals, notified via MQTT, of course.

To raise the nerd factor a bit more, we also decided to remove the built-in goal counter (which needed human interaction count up) and built a very basic small web application which acted as goal counter by using websockets to get MQTT messages when a goal was shot. This web application also implemented the logic when a player has won and published messages to the MQTT broker.

MyMQTT for Android was used as a remote control to reset the game with. With MQTT, of course.

The HiveMQ MQTT broker was the heart of the communication. All communication was done via MQTT with the HiveMQ broker.

Architectural Overview

MQTT Table Football Architecture

MQTT Table Football Architecture

In Action

We brought the football table to an event a few days ago and it was the absolute highlight of the event. Everyone had great fun.

MQTT IRC Bot/Bridge

It has been a long time since my last blog post. I was incredibly busy with HiveMQ and my focus pivoted to M2M in general and MQTT, an awesome, ultra-low footprint protocol for the Internet of Things, specifically. In the future this blog will also cover these things.

I had some spare time this weekend and decided to do some fun programming. The result was a MQTT-to-IRC or IRC-to-MQTT bridge bot. Although it is a fun project, it turns out that there are real useful use cases for that and because of that I decided to share it :)

How to use

The first step is to download or clone my Github Repository. Then, on the command line, simply run:

mvn clean package

Copy the jar file to a directory of choice and create a config.properties file with the properties according to the documentation. An example config:

broker.host=localhost
broker.port=1883
mqtt.clientId=mqttbot

irc.hostname=irc.freenode.net
irc.port=6667
irc.nickName=mqtt_bot_
irc.channels=#myircchannel1,#thesecondchannel

The next step is to install a MQTT broker locally. To do this, go to http://www.hivemq.com and download the latest version. HiveMQ is an advanced enterprise MQTT broker which is made for use cases where scalability, extensibility and reliability is key. It is also perfect for private MQTT projects. Please follow the quick start at here to install HiveMQ.

After installing the HiveMQ MQTT broker locally, start the bot and try publish a MQTT message with a tool of choice on the topic “irc/#%yircchannel1″. Now your message should appear in the corresponding IRC chat.

Of course it is also possible to get all messages via MQTT. Just subscribe to the topic “irc/%myircchannel1/messages” and you should receive all messages of the IRC chat.

Huh? Why should I do that?

At first sight this does not make any sense why one would do that. When taking a second look, you will realize that you could connect any things to a chat with humans. You could send an IRC message when someone enters a door, when your Jenkins has build results, when a Github Commit occurs, and so on.

Also, you may probably want to get a Andorid Push Notification when someone writes your name in an IRC chat. And if you think this does not make any sense, at least it was fun hacking on :-)

Large File Downloads with JBoss Seam 2 and Servlets

Today I worked an an issue which presented users of our application a 403 error page when downloading large files which were generated on the fly. The files were delivered by a servlet and the files were directly copied to the ServletOutputStream, so the files were streamed instead of holding them in memory.

After some debugging it turned out, that the files were not streamed since the Seam Ajax4JSF Filter doesn’t allow to stream files and this resulted in OutOfMemoryExceptions. There are a few ways how to disable or extend the Ajax4JSF Filter (see this Seam Forum Thread).

Since all our download resources are mapped to servlets which resides in the “/media/*” path, it was pretty easy to just enable the Ajax4JSF Filter for all paths except the “/media/*” paths. To achieve this, just add to your components.xml:

<web:ajax4jsf-filter regex-url-pattern="^(?:[^/]|/(?!media/))*$"/>

Set Context Root for a JavaEE 6 Application with JBoss 7.1

Per default, a Web Application which is packaged as a WAR file is mapped to “http://jbossurl:port/war_file_name” when deployed to JBoss AS 7.1. We wanted the application to be mapped to the root context instead of the file name. It turns out, that this is pretty easy to achieve:

  1. Edit standalone.xml in the JBoss configuration directory. Edit the relevant part of the file like this:
    <subsystem xmlns="urn:jboss:domain:web:1.0" default-virtual-server="default-host">
        <connector name="http" protocol="HTTP/1.1" socket-binding="http" scheme="http"/>
        <virtual-server name="default-host" enable-welcome-root="false">

    The important part is the enable-welcome-root=”false”.

  2. Create a file “jboss-web.xml” in the WEB-INF folder of your application.
  3. Add the following to the file:

    <?xml version="1.0" encoding="UTF-8"?>
    <jboss-web>
        <context-root>/</context-root>
    </jboss-web>

Now your Application will be mapped to the context-root of the Application Server and should be accessible with “http://jbossurl:port”.

Programatically Login User with Seam 2 Security Framework

This is a quick one:

When you want to login a user programmatically without checking if the credentials are right or the user is in the IdentityStore with Seam 2, that is pretty easy.


 
final Principal principal = new SimplePrincipal("username");
identity.acceptExternallyAuthenticatedPrincipal(principal);
identity.authenticate();



This will override the Seam 2 Security Framework authentication mechanism and a user with the given username will be logged in, even if the user is not in the IdentityStore of the Application.

I use this regularly when I have some sort of administration interface where I cannot use the default IdentityStore of the application. In this case I provide an own implementation of the required authentication mechanism and just log in the the user with the approach described above. When doing this I add a role to the logged in user (via identity.addRole("myRole")), so I can use all the Seam goodies like the Authorization Annotations (@Restrict) or checks like Identity.loggedIn.

Autostart resque for Gitlab on CentOS 6.2 with RVM installed

In my previous blog post I described how to install Gitlab on a CentOS 6.2 machine. The Resque demon did not start automatically, so I thought I’ll create a little startup script for the resque.sh in the /var/www/gitlabhq directory, so it would autostart. As it turned out, it was harder than expected, because I use RVM. I assume, that you use the installation script I linked in my last blog post to install gitlab, otherwise the following steps could be different for you…. Here are the necessary steps to get it working:

Weiterlesen