Rapid GUI Development for Applications

My first experience of graphical user interfaces for applications was Microsoft Foundation Classes (MFC). MFC really is a horrible API. It's clunky, difficult to learn, buggy, difficult to theme, slow and worst of all, it's not cross-platform compatible. I think the only reason people seem to use it is because it's integrated into Visual Studio.

I have worked on many projects that have attempted to be cross-platform, or have attempted to be as platform unspecific as possible so that an eventual port would be relatively pain free. There are plenty of alternatives to MFC out there. Take a look at this Gui Toolkit site and you will see how many there are. Most of them are implemented in C/C++ and usually comprise an API along with a visual tool for creating the GUI graphically.

These other tool kits range, in my opinion, from the abysmal to the reasonable but there hasn't been one toolkit that really blew me away, was bug free and as extensive as I wanted.

It was while I was working at Virtalis that I spent a decent amount of time thinking about the best kind of GUI for our needs. Virtalis is a company that specialises in advanced visualisation and much of our custom software was 3D based. There was some talk inside the company of developing a 3D GUI - that is, one that would have widgets that were made out of 3D objects. For example, a slider would be a box that could be slid up and down a wire. It's a nice thought and it would be really cool if they decide to implement it one day, but for most of our needs we just wanted a standard 2D GUI toolkit that was cross-platform, easy to use, robust and rapid to develop.

I did a little research and I came up with two great ways of implementing a GUI: one that could be used actually inside our 3D environment; and another that was a standard 2D interface.

Using Flash as a GUI

The first idea was to use Flash/Flex. There is an ActiveX control available that you can use in COM compliant applications but because it's only good for Windows I couldn't use this. Instead I used a cross-platform solution for embedding Flash that was developed by Netscape and is used in Mozilla based browsers like Firefox. It's called NPAPI and although it is documented pretty well, it took quite a bit of detective work to get it working with the Flash plugin (Adobe offer no support).

NPAPI allows you to pass a window handle (type dependent on platform) to the plugin so you can use the window to "bitblt" into a texture. Keeping the technical discussion short, the end result was that I was able to render the output from the Flash plugin into an OpenGL texture. That meant that I could render the texture into an overlay and display it easily inside our 3D environment. It was easy to add interactivity from mouse clicks using the NPAPI but I didn't have time to develop the communications between events in the Flash movie and our application. There are several ways to communicate remotely between a Flash movie and external software using ActionScript3, so even though it wasn't implemented it was possible to do.

The benefits of using Flash as a GUI are numerous:

  • Flex can be used to develop interactive widgets easily from code so GUI development doesn't need a Flash designer.
  • The Flash movie can be rendered on any surface so it can be on the face of a 3D object or rendered to an overlay for a HUD effect.
  • Using Flash means that other cool features like video, sound and animation are easily integrated into a GUI.
  • By encapsulating GUI development inside a Flash movie we enforce separation between the visuals and the document.
  • The solution is cross-platform
  • Using flash movie files allows for modular development and versioning
  • Easily display the GUI in a 3D application, 2D device context or a web browser.

There were also some great ideas I would have liked to play with if I had had time. One would have been to use the alpha channel in the Flash as a means to conveying height data that could be interpreted by a pixel shader. This would be really cool for buttons that you wanted to actually protrude from the 2D surface!

Firefox/XUL as a GUI

If you haven't heard of XUL then basically it's Mozilla's XML markup language similar to XHTML that can be used to easily create common GUI widgets. It's like XHTML forms on steroids! It has a great layout system (unlike CSS) that's quick to learn and concise in syntax. You run a XUL script in Firefox in the same way you would open a web page.

This is all very nice but how do we get this to communicate with our program? This is where XPCOM comes in. XPCOM is similar to MSCOM in that it prescribes a way for components to be created that expose interfaces. (XP stands for cross-platform, not the Windows system.) XPCOM components (different to plug-ins like the Flash plug-ing) can be created in C, C++ and JavaScript. The compiled components (C/C++) take the form of dynamic libraries and are simply copied into a folder in the Firefox installation.

The great thing about XPCOM is that components may be created and instantiated in both C/C++ and Javascript. By writing a C/C++ component, you can instantiate the objects it exposes using JavaScript. You embed JavaScript in the XUL document just as you would a normal web page and effectively call members in the C/C++ component. This is powerful stuff! What this means is that by creating a XPCOM component, you also have a means of controlling it from script - and not just any old script - but JavaScript along with its familiarity and power.

What this means in practice is that you write a component in C/C++ that communicates with your existing system. It could communicate in several ways such as through a socket or shared memory. The interface will depend on what your program does, but instead of writing an interface that has to deal with button clicks, scrolling and such, you write an interface that exposes functions that are more generic and less to do with GUIs. For example, using an existing GUI toolkit I might have to implement a function called OnThrottleIncreaseButtonPressed(), but now I can expose a simple SetThrottle() function and leave the details of how it's controlled to the JavaScript implementation. I can use the slow XUL/JavaScript side for interpreting user input and leave the fast C/C++ implementation to do all the wizzy (simulation, 3D rendering, for example) stuff.

The advantages of this approach are:

  • Once the C/C++ interface/component has been completed GUI development is extremely rapid because you implement the rest in XUL and JavaScript.
  • Change the GUI without the need to recompile in seconds.
  • Create as many GUIs as you like: one for the user, one for the administrator, etc.
  • Use JavaScript to simulate user GUI interaction for unit testing
  • Intersperse GUI widgets with HTML style help text
  • Actually embed your GUI widgets inside your documentation!
  • Utilise browser features to bookmark favourite GUI designs and layouts; use web links to link everything together; or cleverly use the history to undo/redo

There are some drawbacks with using XUL/XPCOM/Firefox as your GUI. Firstly, Firefox is a big application and will draw heavily on the computer's resources. Secondly, some of the development team didn't like the fact that you have to use a web browser. There ways around this though - you can use XULRunner instead of the browser.

Conclusion

The above solutions both heavily utilise existing technologies and will ultimately use up more resources than a normal GUI toolkit would. With the increasing speed of computers I feel that this issue is becoming less important. But the loss in performance, I feel, is more than compensated for in the increase in development times.

By using tools that allow for rapid GUI development it means we change the way the user interacts with our software easily and quickly so to provide the best usability. It means that we can perform usability tests and even change the GUI in front of the customer while they wait.

Web technologies have been around for far less time than application technologies but due to its competitive nature, enormous growth and simple development the Internet has taken usability far more seriously than I feel applications have. By reaching out and using these new technologies for applications we can benefit hugely from exciting projects with endless scope.