Experimental Qt and QML in the browser

Update: You might also be interested in the discussion on Hacker News.

I was amazed when I read Morten Johan Sørvig’s blog post about bringing Qt to Native Client (NaCl) and Emscripten in September last year. I have been following multiple efforts to bring Qt to the web for some time. Mostly because I wanted to make my own Qt projects easily available to others, but also because I find the whole concept of running C++ code in a browser fascinating.

This is definitely not the first time someone has worked on bringing Qt to the browser. Qt4 was ported to Emscripten by Simon St James and published with a lot of examples. However, it seems the port was discontinued when Qt5 came along due to the changes that came with that transition. In the meantime, Lauri Paimen, Anton Kreuzkamp and many others have contributed to qmlweb, a QML interpreter written from scratch in JavaScript. That project does however not make it easy to use any C++ classes you may have defined. And most of my projects are QML/C++ hybrids.

These projects brought something important to the table and took us one step closer to writing QML for the browser. Personally I find QML very enjoyable because of its clean and intuitive structure in comparison to other options like XML. And the ability to easily bind GUI elements to high-performance C++ code has made Qt my favorite framework for many projects.

I tried to port Qt5 to Emscripten myself about a year ago. That turned out to be a real challenge. I based my work on the changes Simon St James made to Qt4 and tried to figure out how to make the same changes in Qt5. After a couple of weeks, I got to the point where I had QtCore running and QtDeclarative (QML) compiling, but other things caught up with me and I had to leave that project behind. The biggest disappointment was that I never got around to making any demos.

I decided to pick up the ball a couple of months ago This time I knew about Morten Johan Sørvig’s efforts and asked him to give me some info on the status of his port. At that point he had support for compiling to NaCl ready (that means Chrome only) and he had already started porting to Emscripten using pepper.js as glue between Emscripten and NaCl. There was still a bit left to do before any examples could be up and running in Firefox. I decided to give it a go and see if I could contribute with something.

A couple of weeks later, I managed to get Qt running in Firefox with a number of QML examples. The process was a lot of fun as I learned much about what happens inside Qt’s sources. It was interesting to see how often I could write many lines of code and rebuild multiple times only to arrive at a one-line change that fixed the issue at hand. I was really happy when I finally could send a patch to Qt’s NaCl branch. The main changes were to Qt’s event loop, which would hang if control wasn’t returned to the browser. I had to add a couple of hacks to keep Qt’s mutexes from locking up everything (this needs to be properly fixed later, perhaps by creating pthread stubs). And OpenGL needed to be explicitly set to OpenGL ES2 in addition to a bunch of smaller modifications.

Below are two resulting demos of QML running in the browser. I’m amazed with the performance both in Firefox and Chrome. Just be aware that the size of the examples is quite big because no optimizations have been done to reduce the size of the binary JavaScript. Further, there may be lots of bugs and some browsers where the examples don’t work. Remember that this is still very experimental and only a small taste of what may be possible with Qt in the browser in the future:

What amazes me the most is the fact that we are now running Qt’s own JavaScript engine, which is written in C++ and compiled to JavaScript with Emscripten, inside the browser’s own JavaScript engine! I really want to see if we can get QML’s WebEngineView running inside itself inside Firefox. That’s a step on the way to true inception, which reminds me of a great talk by Gary Bernhardt titled “The Birth & Death of JavaScript”.

An upcoming C++ library for HDF5 files

I recently started working on a new C++ library for reading and writing HDF5 files. I got the idea when I was working on a few files in Python and C++ at the same time. The Python library h5py is just way more comfortable than the HDF5 C++ API. But with the new features in C++11 and C++14, I figured it should be possible to make a C++ library that is just as easy to use as h5py. And I must admit that I believe I’m on the way to making it even easier.

One design goal I have set for this project is to make the library forgiving. This means that I assume you know what you are doing and try to make it possible, even though there might be some side effects.

One example of such a side effect is that of creating a dataset in a HDF5 file and later changing its size. Because of limitations in the HDF5 standard, the original dataset will still take up space in the file, although it’s no longer in use. I believe the HDF Group designed it this way because it is hard to free already allocated space in a file without truncating it first. So they optimized for performance rather than flexibility, which is a fair choice to make. However, I assume you want flexibility and that you’d rather see that extra space taken up than seeing your program crash for trying to reuse a dataset. So this code is perfectly fine within my library:

#include <armadillo>
#include <elegant/hdf5>

using namespace elegant::hdf5;
using namespace arma;

int main() {
File myFile("myfile.h5");
myFile["my_dataset"] = zeros(10, 15);
myFile["my_dataset"] = zeros(20, 25);
return 0;
}


This opens the myfile.h5 for reading and writing and sets my_dataset to a 10×15 matrix of zeros before resetting it to a 20×25 matrix. This will, however, leave space taken up by the 10×15 matrix used in the file, even though it’s no longer accessible. Oh, and did I mention that the Armadillo library is already supported?

On the contrary, if you try to do the same in h5py, you will get a RuntimeError:

from h5py import *
from pylab import *
my_file = File("myfile.h5")
my_file["my_dataset"] = zeros((10, 15))
my_file["my_dataset"] = zeros((20, 25))
# output:
...


As you can see, the syntax is pretty much the same, but the C++ library will be slightly more forgiving than h5py. That, in addition to a range of other nice features, is what I hope will make this new HDF5 C++ library attractive.

An experimental version will hopefully be released soon.

New project structure for projects in Qt Creator with unit tests

Note: This is a new version of an earlier post, with a revised project structure.

Note 2: See this post for the same project structure using the even better Catch testing framework.

This post assumes that you are using a C++ testing framework such as UnitTest++. See this earlier post for some information on installing UnitTest++ on Ubuntu.

To get the most out UnitTest++ it is a good idea to integrate its output into the Qt Creator IDE. The way I have set this up in Qt Creator is with subprojects. One for the main project, which again is split into the app itself and a library, and one for the tests. In addition, I have a helper project file, named defaults.pri. The structure of the project is like this:

MyProject
├─ MyProject.pro
├─ defaults.pri
├─ app/
│  ├─ app.pro
│  └─ main.cpp
├─ src/
│  ├─ src.pro
│  └─ myclass.cpp
└─ tests/
├─ tests.pro
└─ main.cpp

An example project using this code structure has been posted on Github by Filip Sund. (Thanks to Filip for doing this!)

The main project file, MyProject.pro will now be based on a subdirs template, and may look like this:

TEMPLATE = subdirs
CONFIG+=ordered
SUBDIRS = \
src \
app \
tests
app.depends = src
tests.depends = src

The app.depends and tests.depends statements makes sure that the src project is compiled before the application and tests, because the src directory contains the library that will be used by both the app and the tests.

(Thanks to Will for noting that this works better for parallel builds with make -j 8 than my previous version only using CONFIG+=ordered. We should keep CONFIG+=ordered in there still, though, because depends doesn’t affect the order when using make install).

defaults.pri

Each of the other .pro files will include defaults.pri to have all the headers available. defaults.pri contains the following:

INCLUDEPATH += $$PWD/src SRC_DIR =$$PWD

If the library, main program and tests use common libraries, it is very useful to have the defaults.pri define these dependencies too.

./src

In the src folder, I have myclass.cpp, which is the class that I want to use and test. The src.pro needs to compile to a library, so that it may be used both by app and tests, and could look something like this:

include(../defaults.pri)
CONFIG -= qt

TARGET = myapp
TEMPLATE = lib

SOURCES += myclass.cpp
HEADERS += myclass.h

What this class does is not so interesting, it could be anything. A simple example could be this header file:

#ifndef MYCLASS_H
#define MYCLASS_H

class MyClass {
public:
};

#endif // MYCLASS_H

With this accompaning source file:

#include "myclass.h"

double MyClass::addition(double a, double b) {
return a * b;
}

./app

I only have a main.cpp file in app, because the app is basically just something that uses everything in the src folder. It will depend on the shared compiled library from src, and app.pro would look something like this:

include(../defaults.pri)

TEMPLATE = app
CONFIG += console
CONFIG -= app_bundle
CONFIG -= qt

SOURCES += main.cpp

LIBS += -L../src -lmyapp

The main.cpp file could be a simple program that uses MyClass:

#include <myclass.h>
#include <iostream>

using namespace std;

int main()
{
return 0;
}

./tests

In the tests folder I have simply added a main.cpp which will run the tests. Then tests.pro has the following contents:

include(../defaults.pri)
TEMPLATE = app

CONFIG   += console
CONFIG   -= app_bundle
CONFIG   -= qt

SOURCES += main.cpp

LIBS += -lunittest++ -L../src -lmyapp

Which now links to the myapp library in addition to the unit tests.

The main.cpp in tests file which could contain the following, if we were to use UnitTest++ as our testing library:

#include <unittest++/UnitTest++.h>
#include <myclass.h>

TEST(MyMath) {
MyClass my;
}

int main()
{
return UnitTest::RunAllTests();
}

This test will fail because my implementation of MyClass::addition is completely wrong:

class MyClass {
public:
double addition(double a, double b) {
return a * b;
}
};

Note that I’m including MyClass by through <myclass.h> which is possible because of the INCLUDEPATH variable in defaults.pri.

This is hopefully all you’ll need to define a project that compiles a library, as well as tests and an application using the library.

Writing molecular dynamics data to binary LAMMPS format

In this post I will explain  how to write to the binary LAMMPS file format from C++, using data stored in Armadillo vectors and matrices. After running the example in this post you should be able to open the resulting file in Ovito or any other program capable of reading binary LAMMPS files. The example should also be fairly easy to port to other data structure type, if needed.

For the impatient: You’ll find a working main.cpp file and a qmake project file on GitHub.

The result, if rendered in Ovito, is two silicon atoms (in red) and one oxygen atom:

LAMMPS is a molecular dynamics simulation package that is extremely versatile with plenty of interaction potentials and features implemented. However, you may have written some other code involving atoms and found yourself in the position of considering using a standard file format to write atom data to file. In this case, the XYZ-format has likely passed your mind, but because this is a ASCII-based text format, it is slow to read and write. This format also lacks standardized headers for information such as the system boundaries – causing visualizers like Ovito to have to guess for the right boundaries in your system.

Using Qt3D today

Qt3D is an amazing library for Qt that gives you the ability to render your own 3D stuff together with your existing widget or QML based GUI. The library was started by the Qt developers a few years back, but has not yet been released with the official Qt SDK. It was announced that it would be bundled with Qt5.0, but because of the state of the library at the time, its release was postponed. Even so, the library is still very mature and appears to work very well in my opinion. So I would urge you to test it out.

Simple install on Ubuntu

To use the library you have a couple of different choices. If you are running Ubuntu, you may just run the following commands:

sudo apt-get install qt3d5-dev qtdeclarative5-qt3d-plugin

And you should be good to go. Look for examples in the Qt3D documentation.

Note that Ubuntu currently doesn’t have QtQuick.Controls available, so you will have to follow the instructions below to combine Qt3D with QtQuick.Controls.

Install Qt3D into the newest Qt release

Currently Qt5.1.1 is the newest stable release of Qt and is the version I’d recommend that you use with Qt3D.

2. Clone Qt3D from Gitorious:
git clone git://gitorious.org/qt/qt3d.git qt3d
3. Open QtCreator from your new Qt installation.
4. Open the qt3d.pro file from the cloned repository and configure it to use the platforms on which you want to use Qt3D (desktop, Android, etc.).
5. Build for each platform you want to use. Set the build to Release instead of Debug mode while building. You should receive about one hundred warnings if you are building for Android. As always, ignore these.
6. Open the build folder in a terminal and run
make install

This will not install to your system, but to your Qt installation, so that it gets the ability to do awesome 3D stuff.

You should now have Qt3D installed in your Qt installation and be ready to run projects that are using Qt3D.

Monitoring your unit tests without lifting a finger

I love unit testing. First of all, I think it is a good idea to test separate units of the code, but after doing so for some time, I’ve come to realize that unit tests are great for managing the software development cycle too. It all boils down to the idea that you should write tests before you write your code.

Now, this is something that I and others apparently struggle a lot with. How do you write a test for some code that doesn’t even exist yet? Even worse, how do you write a test for a piece of software that you’re not yet sure how will be used?

In computational physics, this problem arises often because we are writing code at the same time as we are trying to understand the physics, mathematics and algorithms at hand. And this is a good thing. You might want to think that one should structure all code before it is written, but this is generally a bad approach in computational physics. Especially if you’re working on something new. The reason is that you will often understand the problem and algorithms better while developing, rather than just reading about them and trying to analyze them blindly.

Keeping the tests and code healthy

But enough with the talk, let’s just assume that you are convinced that you should (or have to) implement some unit tests. At one point you are likely to be in a position where you find it tiresome to have to go into that folder where the tests are defined and run them manually. This is where Jenkins comes in to play.

Speeding up compilation on Ubuntu with Qt Creator

Are you reading random stuff on the web while waiting for your C++ compilation to finish? Then you have come to the right place. In this post I will tell you about two really nice tweaks you may do to speed up your compilations, namely ccache and the make -j flag, and how you may set these up in Qt Creator.

ccache

ccache is a clever tool that wraps you compiler (g++ or mpicxx) and understands whether the file you are compiling has been compiled before with exactly the same contents and settings. If it has, it just returns a cached result.

Unlike regular make, ccache is extremely good at detecting the true state of what you are compiling. I have never had any trouble with ccache.

This really speeds up the compilation when you are using make clean, especially if you are switching git branches. In other words, it is a much simpler solution to achieve fast compilation with git branches than to create separate build folders for each branch.

To enable ccache, install it with

sudo apt-get install ccache

QMAKE_CXX = ccache g++

Replace g++ with mpicxx if you are using MPI.

The make -j flag

I realized when compiling the Qt source that make has a -j flag that enables threaded compilation on all available processors on the machine. This also speeds up compilation significantly, and I made a 3.55x performance gain on a 4 core CPU. To enable this flag, go to the Projects view in Qt Creator and add the following arguments to the make build step:

-j

This should look something like this afterwards:

If you prefer not to use all available processors for compilation, you may add a number after -j to set the number of processors. For instance make -j 3 would compile with 3 processors.