AS OPERATING SYSTEMS increasingly become visual feasts, those who want to create useful interaction enhancements are having to bend over backwards thanks to closed source software in order to bring innovation to the user's environment.
Two bright young men from the University of Washington recently presented Prefab, a technology which they say will facilitate the implementation of "advanced behaviours in graphical interfaces". That in itself isn't particularly new but the route Prefab takes to implement well documented graphical user interface (GUI) techniques are a clear example of the lengths engineers have to go to circumnavigate the limitations posed by closed source software.
Given that Prefab, the system coined by Morgan Dixon and James Fogarty, has no access to the pre-compiled source code, the system effectively reverse engineers the GUI. That reverse engineering is merely the first of a three step process, allowing the system to overlay enhancements irrespective of the program or operating system and relay inputs back to the original "source" GUI.
To understand how Prefab works, one must look at how companies such as Apple and Microsoft put increasing demands on developers through the use of various APIs and toolkits to produce applications with similar 'look and feel'. The perceived visual affinity between applications helps breed familiarity and branding. Despite the largely symbolic link that exists on the surface, useful behavioural enhancements require much more than just a coherent colour scheme.
Advances in human computer interface are rarely the headline technologies such as Apple's Multitouch, Microsoft's Surface or even Xerox's PARC User Interface. These days, the majority of developments are far more incremental and aim to improve the user experiance by combining a number of what may seem insignificant technologies.
The researchers say that their system provides reliable and reproducible results due to the very "consistent look and feel" that applications have due to the reliance on development toolkits used to produce GUIs. Those toolkits typically have the same dialogue boxes, buttons and other "pre-fabricated" components used to create the GUI.
In order to create a system which is software agnostic, Prefab goes down to the most basic GUI element, the pixel. It performs a "pass" over the image to identify any potential areas or "hotspots" where enhancements can be applied. The pixels are then fed into a decision tree and determine whether the hotspot matches any known patterns in a generated library.
After confirming whether a particular area of the screen requires enhancements Dixon and Fogarty show the application of interface enhancements such as Bubble Cursor, Sticky Icons and Phosphor, and how input from the modified fields is passed back to the program, maintaining its integrity despite being reverse engineered.
It is the application of Phosphor, however, which is the most impressive. This enhancement, which produces an afterglow on, for example, a slider, was initially unveiled back in 2006 yet still remains difficult to implement. The difficulty lies in the method requiring almost instantaneous feedback to update the afterglow. Given the level of computation that has to take place this poses serious efficiency questions on the algorithm, questions which Prefab manage to answer, in the demonstration.
However the most impressive aspect of Prefab is its ability to transfer these techniques to non-standard operating system widgets. The researchers show the enhancements on a Youtube video player as an example of 'non standard' GUI buttons. The coup de grace for Prefab is its ability to extend its agnosticism by working on different operating systems. This is demonstrated by using VNC to connect to a machine running OS X and applying the same GUI enhancements, in real time, that are present on the local system with no modification.
However the sterling work being done by researchers such as Dixon and Fogarty, is being undermined by companies unwilling to embrace open source. One such "promising" enhancement which has yet to take off is the Side Views parameter spectrum. The technique simultaneously previews options for values of multiple parameters, similar to colour charts which show an automatic preview as the red, green and blue sliders are moved.
In order to demonstrate the technique, Dixon and Fogarty were forced to use the GNU Image Manipulation Program (GIMP) rather than Adobe's Photoshop because the researchers say "the open source implementation [of GIMP] allowed necessary modifications" and added that the closed source Adobe program "prevents researchers and practitioners from studying or deploying" techniques.
The problem isn't limited to just the parameter spectrum technique or Adobe Photoshop, with the researchers, absolutely correctly, calling out the way closed source software limits their innovation.
The idea that having closed source software somehow helps developers protect their investment is merely a short sighted view of the software development cycle. While protecting the source code may keep past achievements safe, the inability to harness the talent of researchers and the open source community is not only hurting their product but, ultimately, the users.
Prefab is simply one example of the lengths that innovators such as Dixon and Forgarty have to take in order to implement well documented and established technologies. Reverse engineering down to the fundamental display unit, the pixel, is impressive but ultimately inefficient.
But the inefficiency isn't due to poor quality work on the researchers' part. Rather, it stems from an endemic reliance by commercial developers on restricting access in order to protect their own short term business profits rather than taking a more enlightened approach to their long term business interests, not to mention serving the greater good for the industry and customers.
Prefab is an example of what can be achieved in open source projects such as Firefox and Linux, innovation without the barriers imposed by short sighted commercial greed. µ
Innovation over elaboration?
How IT is being used to screw democracy around
But Brexit means the UK probably won't be affected
But Microsoft still denies culpability