
\part{Conclusions}
\label{chp:conclusions}

In this thesis we have tackled three real-time computer vision problems, all of which are motivated by their potential application to vision-based computer games. This motivation stems from an industrial collaboration with Sony Computer Entertainment Europe, who are interested in using computer vision to provide a platform for the development of modern and accessible computer games. The desire to tackle problems and produce techniques which have real-world applications has been an important factor throughout this thesis, and we hope that the approaches that have been presented can provide building blocks for future research and product development in this space.


% An interesting and sometime challenging factor has been the desire to satisfy both academic challenges, whilst bearing in mind the desire to make useful real-world contributions. 

A common theme throughout this thesis has been a focus on computational efficiency, since gaming applications typically demand real-time algorithms which can be run interactively as frames are received from a camera. This requirement has influenced many of the design choices which have been taken when developing solutions in this work. Furthermore, the work in Chapters \ref{chp:learnmatch} and \ref{chp:reconstructor} has focused on providing solutions which are suitable for low-powered devices, which are an increasingly important platform from a gaming perspective. While the power of these devices is increasing at a rapid pace, they still possess only a fraction of the power of a typical desktop computer, meaning designing real-time computer vision algorithms for them still presents a major challenge.

The other major theme throughout this thesis has been online structured learning, which has been incorporated into all of the solutions we have developed. We have used online learning in order to provide a principled and computationally efficient means for incorporating adaptability into our algorithms, which is essential for handling the wide variety of environments we expect to encounter when deploying vision-based games in the real world. Incorporating structure into the learning results in even greater gains, since the learner is more tightly integrated into the overall pipeline, meaning the adaptability is focused correctly for the target application.

% all of the problems we have tackled have been focused on real-time applications

% furthermore, we have been particularly interested in low-powered devices, since this is a growing platform for augmented reality gaming

% All of our solutions have made use of online learning, which provides a principled and efficient way of incorporating adaptability into our approaches.

% Furthermore, we have used structured learning, which has various benefits: learning more integrated, ...


%\pawanchapter{Contributions}

% summarise chapter by chapter?


\chapter{Contributions}

In Chapter \ref{chp:struck} we considered the task of 2D arbitrary object tracking, which has many potential applications for human-computer interaction and AR. We presented a novel approach for this task which makes use of online kernelised structured output learning in order to model the appearance of the target object during tracking. Our method is able to adapt online to appearance changes of the target object and its surrounding background during tracking, and does so using a principled structured learning framework which takes the entire tracking pipeline into account, rather than artificially introducing an intermediate classification stage. The use of kernels provides great flexibility in terms of the image representation which can be used by our method, allowing different image features to be used and combined together. We also introduced a budgeting mechanism which ensures that the computational complexity of our approach remains bounded, meaning it is suitable for the real-time applications we are targeting. Experimentally, we observed that our framework results in a tracking algorithm which delivers state-of-the-art performance on standard tracking datasets.

Chapter \ref{chp:learnmatch} continued the theme of adaptive object tracking, this time focusing on keypoint-based object tracking, which is central to many AR applications. The approach we presented takes the traditional pipeline of keypoint matching and geometric verification, and embeds this within an online structured learning framework. In doing so, our approach is able to provide a principled mechanism for adapting the detection pipeline for a specific object and background environment. This allows our approach to provide significant improvements to detection performance compared with traditional methods and means we can handle challenges such as repetitive features and confusing background, which we demonstrated experimentally. Our approach adds only a small amount of overhead compared to a non-adaptive approach, and we further showed how we can make approximations which allow us to take advantage of recently proposed binary keypoint descriptors, allowing for real-time operation even on low-powered devices.

In Chapter \ref{chp:reconstructor} we tackled a different problem related to AR: scene reconstruction for SLAM on low-powered devices. In common with other work in this thesis, we framed the task as one of structured prediction and presented an approach which is able to automatically identify a small number of dominant planes in a scene, along with estimates of their boundaries. To perform this boundary estimation, our approach initially makes use of simple multi-view photo-consistency information, and subsequently incorporates online learning of the appearance of each plane to help refine the reconstruction. The resulting algorithm is computationally efficient, and we show that it is able to run on a typical scene in a few seconds on a low-powered device, making it suitable for mobile gaming applications based around SLAM.


\chapter{Future work}

When developing the approaches described in this thesis, there was a desire to produce principled frameworks which could be built upon by future research. The use of online learning, in particular, means there is a great deal of existing research from the machine learning and computer vision communities which could be incorporated into the approaches we have presented in this thesis.

For the 2D object tracking approach presented in Chapter \ref{chp:struck}, future work could include extending the output space which is used during tracking. One example would be to consider tracking which takes into account object deformation and articulation, while another would be to handle jointly tracking multiple target objects. Other potential avenues include exploring different types of image features, as well as incorporating online multiple kernel learning \cite{Gonen2011} to choose features which are well suited to a given object and environment. Another interesting direction would be to adapt this algorithm so that it is better-suited to low-powered devices, perhaps by using binary features such as those used for keypoint matching in Chapter \ref{chp:learnmatch}.

For the keypoint-based object tracking approach presented in Chapter \ref{chp:learnmatch}, our approach is already able to down-weight those keypoints which are less discriminative in order to better detect the target object. However, we do not perform feature selection explicitly, so future work might include adding a sparsity-inducing norm \cite{Tibshirani1996} during learning to explicitly encourage feature selection. Another interesting avenue could be to model and learn keypoint deformation, which would allow the tracking of deformable and articulated objects. However, this would be difficult to achieve in an online learning framework which uses self-training, as measures would need to be taken to avoid drift.

For the scene reconstruction approach presented in Chapter \ref{chp:reconstructor}, as has already been discussed in Section \ref{sec:reconstructor_conclusions}, there are a number of avenues for future work which would help to improve the reconstruction results. These include how best to handle the background class during labelling, improvements to labelling accuracy, and fusing multiple 2.5D reconstructions into a global 3D reconstruction.

\vspace{0.7cm}

\noindent One thing which is clear is that the increasing ubiquity of portable, powerful, devices containing cameras makes this an exciting time for the field of computer vision in general and presents many opportunities for vision-based gaming in particular. We hope that the work presented in this thesis will have contributed some building blocks which can be built upon by others both in academia and in industry.


%When developing the approaches described in this thesis, there was a desire to produce principled frameworks which could be built upon by future research. This is particularly true for the approaches described in Chapters \ref{chp:struck} and \ref{chp:learnmatch}, which both use online structured learning in order to provide frameworks for adaptive tracking, in which the learning is tightly integrated into the resulting pipeline.
% change 