1.0 Introduction
The MultiScreenToolkit evolved out of several european projects that we (Noterik) are in and provide backend/frontend services for (read european projects page for more info) during these projects we found a need to be able to solve the following problems :
- How do we handle multiple formats (single/multiscreen, group viewing)
- How do we handle multiple devices (phones, tablets, tv and HbbTV)
- How can partners in the project be involved in the development
- How can we host applications built using the SMT in the cloud
- How do we display the information streams in different modes
Within the toolkit we tried to resolve most of the problems outlined above and create a solid base for use in these projects and we hope also for your usecase. We will do our best to outline the concepts behind the solution we came up with in the next few pages but if you feel a more visual explanation fits you more visit our youtube channel where we try to cover the same topics in a different form.
Instead of having different solutions for single screen and second screen applications we developed a concept where an application is independent of how many screens are involved and can in fact dynamically react to changes in the amount and types of screens attached to the application. In concept the idea is simple but we found out it requires people to make shift in the way they think about how they view these types of applications.
The big change is to go from worldview that everybody opens/starts their own application to that viewers join an application already running on the server (image below). This is a more modern way of thinking similar to joining for example a google-doc 'page' (application) or your facebook 'page' (application).
This sounds like a slight change but has huge effects on both what we can do and the technical implementation since now the application is able to see and interact with all the users in a unified way. From the developers point of view this now means that there is only one application that is always running (even if there nobody is watching) and that screens, users and devices changes are all just signals to this running application that it can and has to react on. This way it becomes possible for an application to grow from providing content from one screen to many when more users start watching and adapt to what each screen is capable of.
Creating this unified model for the developer does of course mean that the underlying framework (the toolkit) has to resolve many technical communication and synchronization problems since in reality the application is not fully running inside the server but involved many HTML5 browsers at clients, HbbTV clients, remote controls and even gestural interfaces. We will expand on this later in this document but from the unified application view this level of abstraction is actually the goal developers should not have to worry about how its done but should be able to rely on the framework to resolve it.
It’s became clear during the development of our projects that we need not just need to support desktops and/or HbbTV but in reality its a mix that also includes smartphones and tablets. To keep the system as flexible as possible we decided to focus on HTML5 based applications with a downscaling to HbbTV specs 1.5 and possibly 2.0 in the future. But within this limitation we again feel it’s important that we can have a unified view for the developer and that the framework handles as much as possible in matching the device capabilities to our internal model.
Traditionally one would use adaptive/responsive design using stylesheets to adapt to different screen sizes and browser. And we use this on the lower levels inside the HTML5 clients but we also need a more generic concept on the application level. From the applications point of view each client (mostly browsers) are just seen as screens with capabilities and this is the abstraction level we want to support.
We distinguish between the following screen capabilities:
- size (pixel count and physical size)
- rotation (is the user holding it in landscape or portrait)
- stylesheet level (what options for visual styles do we have)
- passive or active (is it just a screen or also provides mouse, touch input)
- keyboard support (is there a fixed or optional keyboard)
- brand specific features (special buttons for example)
Each of the screens works with these capabilities as almost a contract (interface) between it and the system. Communication from and to these unified screens is either handled by the framework as part of the abstraction concept (so for example if a state of a button changes in the application the framework will change it on all the screens needed) or is done manually using a simple message back to the server side of the application (image below).
Using these methods we currently support all HTML5 based desktop browsers, android phones and tablets and Apple (iOS) phones and tablets.
One of the biggest problems we ran into over the last years is the wish for a more open development process for the frontend applications. With the focus to make it easer for new and external developers to create applications on top of our framework (Springfield).To enable this we continue the concept of splitting the framework and applications, where as outlined above the framework takes away as much of the groundwork as possible to keep developing for these complex environments. As a side effect it became clear that it would now be possible for others to also create applications without being involved in the development of the framework and more important the hosting platform we provide for these applications. The design we followed is the traditional model used for applets, servlets, restlet and other programming systems.The name we picked as base class is ‘html5application’ (image below).
With most of these programming concepts also comes a development model, tools for packaging, distribution and hosting. Clearly it would be out of the scope of this project to invent such a tool chain for our applications so we decided to coopt a well known model that people already know and use in the community and added some special parts to make it work for us. The system we picked was the J2EE [3] programming model that is already used for the Springfield [4] system itself. Developers can program using any J2EE toolchain (we use Eclipse and Tomcat/Apache) use the normal compile and deploy tools (using the jar and war standards). The main difference is that unlike normal wars that need to be deployed to a J2EE application server you now deploy the application to a newly designed deployment area inside a Springfield cluster (image below). Partners can still decide in the future to host their own Springfield cluster but they now also have the much easier and better understood model of just writing applications and deploying them on the already available cluster inside the LinkedTV project.
