
\part{Planar Scene Reconstruction for Portable SLAM}
\label{chp:reconstructor}
%\lhead{Chapter 1. \emph{Struck: Structured Output Tracking With Kernels}} 

%\TODO{Chapter \ref{chp:reconstructor} continues the theme of augmented reality on low-powered devices from Chapter \ref{chp:learnmatch}, and presents an approach for approximately recovering the 3D structure of a scene which has been mapped by a simultaneous localisation and mapping (SLAM) system. Our approach is guided by the constraints of the hardware we are targeting, and we develop a system which coarsely models the scene using a small number of planes. In common with the work in other chapters, we introduce online learning into our approach, and show how a relatively simple image segmentation framework coupled with online learning of appearance models can produce reconstructions of a scene which are useful from a gaming perspective.}


% chapter intro: we have already seen approaches for tracking, we now move to another application useful for computer games: scene reconstruction. once again we are interested in a solution which works in real-time, or close to real-time, and suitable for low-powered devices. in this chapter we present a simple approach based on planar reconstruction. as in previous chapters, we show how we can use online learning to help with this task, in this case by learning appearance information about each plane.




%We're specifically interested in doing this for a low-powered device, so our approach is designed from the ground up to be suitable for such a device. 




\chapter{Introduction}

The work presented in this chapter continues with the theme of augmented reality on low-powered devices from Chapter \ref{chp:learnmatch}. We now turn our attention to the task of scene reconstruction for mobile AR gaming based upon simultaneous localisation and mapping (SLAM). Tackling this particular problem is motivated by the industrial collaboration with Sony Computer Entertainment Europe, as this is an area which has been identified as being of particular interest in the context of making vision-based computer games. The work in this chapter can therefore be seen in a slightly different light to those preceding it, since it is concerned with attempting to produce a practical solution to a specific real-world problem given certain constraints. Nevertheless, the approach which we present here is closely linked to the work in previous chapters as we treat the task as one of structured prediction and show that the addition of online learning into the resulting framework can help to improve the quality of the resulting reconstruction algorithm.

% mention that unlike previous chapters, this is more applied? not really trying to improve over other baselines, but rather trying to solve a real problem and produce a working system to a particular problem.

Most current AR applications make use of a known target object which can be detected and tracked in 3D using the keypoint-based approaches discussed previously in this thesis. As well as being used as a tracking target, the physical object typically then provides a `stage' upon which virtual content can be displayed such that it appears realistically in the scene. Such an approach means that the AR experience requires the user to have this physical object in front of them, for example an image in a magazine or on product packaging.

Recently, a great deal of progress has been made in the field of vision-based SLAM, and there are now a number of robust approaches \cite{Klein2007,Davison2007} which can be employed to reliably track the 3D pose of a camera in real-time as it moves in a previously unknown physical environment. A lot of subsequent engineering effort has also gone in to allowing these approaches to run on low-powered devices such as smartphones and portable games consoles.

SLAM has the potential to provide a powerful platform for AR gaming, as it is able to map large physical areas and, importantly, does away with the requirement that the user has a known object in front of them. This gives much greater flexibility in terms of when and where an AR experience can take place. However, it has the associated drawback that there is no longer a known stage on which to place virtual content. The goal of the work in this chapter is thus to develop a system able to provide a reconstruction of the underlying scene as it is explored, such that virtual content can be displayed in a realistic manner.

The majority of approaches to SLAM are based around sparse representations of the scene. The map which is built by these systems consists of a set of distinctive 3D keypoints which can be reliably tracked, which are then used in order to estimate the 3D pose of the camera in each frame. While this sparse representation is sufficient for the task of camera tracking, and has the benefit of being computationally efficient, it does not generally provide enough information for the higher-level task of displaying virtual content in the scene. For this purpose, a more complete reconstruction of the scene is required.

In this chapter, we develop an approach which uses a sparse SLAM system running on a low-powered portable games console as its starting point and aims to produce a simple reconstruction of the scene. Our target application is tabletop AR, in which we envisage the user having a playing surface along with some other objects such as boxes or books.
%picture
Guided by this application area and the constraints we have in terms of computational power, we propose modelling the scene using a small number of planes, the boundaries of which we then attempt to estimate using cues from the input image stream. Besides the computational benefits, modelling the scene in this way has additional advantages for gaming applications, as the resulting reconstruction is more semantically meaningful than \eg\ a mesh, since each planar region defines a distinct area on which game content can be displayed.

% in SLAM context, we do not need reconstruction to be performed every frame. we just need it to be quick enough that it can give a reconstruction online e.g. at the start of the game. 

In common with the work presented in other chapters of this thesis, the approach we develop is framed as a task of structured prediction. We formulate scene reconstruction as a pixel-wise labelling problem and use a CRF to impose structure on the solution. We use relatively simple multi-view photo-consistency information in order to keep computational requirements low, but show how we can also incorporate online learning based on the appearance of each plane in order to refine the initial solution. In this way, our approach results in an efficient reconstruction algorithm which we demonstrate to be suitable even for low-powered devices.

%One simple approach which can be employed is to instruct the user to initialise SLAM on a planar area such as a tabletop or wall, which can then provide a planar stage on which to display content. However, this approach is overly restrictive, since the underlying SLAM system is capable of mapping much richer 3D environments. 

% maybe show a figure of how AR can look bad when e.g. things go off of a table, or through other objects. 


\chapter{Motivation and related work}

Multi-view reconstruction has a rich history in computer vision and many sophisticated approaches have been proposed. Algorithms are typically provided with a set of calibrated images of a scene from multiple viewpoints and then proceed to infer 3D information about the scene using multi-view stereo \cite{Seitz2006}. The calibration information can either come from a carefully-controlled capture environment in which the 3D pose of the camera is known in advance, or by using structure-from-motion techniques \cite{Hartley2004} to recover the calibration information from the images themselves. These approaches are typically designed to operate offline, with the goal of producing highly accurate reconstructions, without particular concern for computational constraints.

A SLAM system is itself performing real-time structure-from-motion and is therefore able to provide a set of calibrated images suitable for multi-view reconstruction. Because of this, there has recently been research interest in adapting multi-view reconstruction algorithms to a real-time SLAM setting. These methods are based on traditional reconstruction techniques, but take advantage of the fact that some of these algorithms lend themselves well to parallelisation. This means that these approaches can be implemented using general-purpose graphics processing unit (GPGPU) programming and make use of the extremely powerful graphics hardware in modern computers. In this way, the approaches developed by Newcobe and Davison \cite{Newcombe2010,Newcombe2011} and Stuehmer \etal \cite{Stuehmer2010} are both able to produce highly-detailed dense reconstructions of a scene as it is explored by a handheld camera.

The goal for our own work is also to produce a scene reconstruction in real-time using the result of a SLAM system, but we are specifically interested in doing so on low-powered portable gaming devices. In this setting, even performing SLAM using a sparse representation presents a significant computational challenge and requires careful engineering. Furthermore, GPGPU programming is not an option, since the limited graphics hardware available is entirely used for displaying game content, meaning any solution must be suitable for a low-power CPU.

% mention the Antony dick work. Performs 3D reconstruction of an object based on silhouette consistency. however, the approach requires user interaction, and is only suitable for a single object

Given these constraints, the approach we take is to simplify the reconstruction task by modelling the scene using a small number of planes. While this assumption will not be suitable for all scenes, for the application of tabletop gaming which we are targeting we expect it to be able to capture the coarse structure of the scene adequately.

Similar piecewise-planar modelling approaches have been previously used for multi-view reconstruction, particularly in the case of urban street scene reconstruction, where the goal is to reconstruct building facades and roads \cite{Gallup2010,Sinha2009}. The motivation for using a planar assumption in these cases is that although it provides a simplified reconstruction of the scene, the complexity of the resulting model is constrained, meaning it provides a form of regularisation of the solution and can better handle challenges such as poorly textured or specular surfaces. These approaches are not designed for real-time operation on low-power devices, however, and still require significant computational resources.

% real-time plane-fitting approaches for SLAM: ninja-on-a-plane, discovering higher level structure in visual slam. but these fit rectangular planes only, so not good for finding accurate boundaries

The contribution of this chapter is a structured prediction framework for performing fully-automatic coarse scene reconstruction on a low-powered device in a few seconds, meaning it can be used in conjunction with a SLAM system running on this device to provide a platform for AR gaming. We also demonstrate that by introducing online learning of appearance information into our framework, we are able to refine the initial reconstruction solution obtained from photo-consistency information alone. In this way, our approach is able to adapt to a given scene and better handle situations in which photo-consistency is not informative or reliable.

%contributions:

% a reconstruction algorithm targeted for low-powered devices
% the addition of online learning to help refine the reconstruction

\chapter{Our approach}

\section{SLAM system}

% overview of the SLAM system: keyframes, landmarks, measurements

The starting point for our method is the \emph{Magnet} SLAM system developed by Sony Computer Entertainment Europe, which has been designed to run on the PlayStation Vita portable games console. This system is based on the PTAM method proposed by Klein and Murray \cite{Klein2007}, but has been carefully engineered to allow it to run on a low-powered device. As the camera explores a scene, the system constructs and maintains a \emph{map} consisting of a set of 3D \emph{landmarks} $\mathcal{L} = \{\vt{p}_1, \dots, \vt{p}_{N_L}\}$, along with a set of \emph{keyframes} $\mathcal{K} = \{K_1, \dots, K_{N_K}\}$, where each keyframe is a tuple $K_i = (I_i, \mathcal{M}_i, T_i)$ consisting of a $320\times240$ pixel RGB image $I_i$, a set of 2D image-space \emph{measurements} of a subset of landmarks (\ie\ those landmarks which have been successfully tracked in $I_i$) $\mathcal{M}_i$, and the estimated 3D pose of the camera $T_i$. Over time, new keyframes are added to the map and a background thread periodically performs bundle-adjustment \cite{Hartley2004} in order to jointly refine the estimates of the landmark positions and keyframe camera poses. For performance reasons on a low-powered device, both the number of landmarks and the number of keyframes are kept relatively low, meaning the maps built by this system are particularly sparse. For a typical tabletop scene, we can expect something of the order of $N_K = 50$ and $N_L = 200$. Figure \ref{fig:reconstructor_landmarks} shows an example of a typical tabletop scene, along with the landmarks which have been inserted into the map.

One factor which is difficult to handle in a SLAM system is scene scale, since this can not be directly estimated using visual information alone. While it is possible to estimate true scene scale given additional sensor information from accelerometers and gyroscopes, this functionality is not present in the \emph{Magnet} SLAM system. Instead, the scene scale is arbitrary, with the initial landmarks inserted into the map at an average distance of 15 units from the first keyframe. The fact that scale is unknown does not cause problems in practice, particularly since for the tabletop scenes we are interested in, the true scale of the scene stays roughly constant.


% discuss colour space (Lab probably)?
% discuss scale?

\section{Plane finding}
\label{sec:reconstructor_plane_finding}

% finding planes given the sparse point cloud

The sparse landmarks tracked by the SLAM system correspond to locally planar surface patches in the scene. The first stage of our approach aims to automatically identify larger planes which are supported by clusters of these landmarks. Our assumption is that each landmark can either be assigned to one of these larger planes, or otherwise can be labelled as part of the `background' of the scene, meaning it does not belong to any plane. To achieve this, we make use of the energy-based model-fitting approach PE\textsc{a}RL \cite{Isack2011}. In essence, this approach offers a means for performing RANSAC \cite{Fischler1981} for fitting multiple models. However, by fitting these models simultaneously, rather than greedily fitting them individually, it has been shown to produce superior results \cite{Isack2011}. Crucially, the method also offers a means for automatically estimating the appropriate number of models to fit, which is essential for our application. 

\begin{figure}[tb]
\centering
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=\textwidth]{images/landmarks.png}
\caption{Landmarks}
\label{fig:reconstructor_landmarks}
\end{subfigure}
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=\textwidth]{images/planes.png}
\caption{Plane assignments}
\label{fig:reconstructor_planes}
\end{subfigure}
\caption[A typical tabletop scene]{A typical tabletop scene. The left image shows the landmarks which have been inserted into the SLAM map, while the right image shows how these landmarks are automatically assigned to planes using the approach described in Section \ref{sec:reconstructor_plane_finding}. Here 4 planes have been identified, each of which has a different colour, while red corresponds to the background class.}
\label{fig:reconstructor_example_scene}
\end{figure}

We begin by generating an initial set of plane hypotheses $\mathcal{H}$, where each hypothesis is described by a parameter vector $\theta_h$, consisting of a 3D normal vector and the distance from the origin. To generate $\mathcal{H}$ we perform a 2D Delaunay triangulation of the landmarks when projected into a single keyframe (the choice of keyframe is arbitrary, in practice we use the reference keyframe discussed in the following section). We use each resulting triangle to define a plane hypothesis, by computing the plane passing through all three landmarks in 3D. We also include an additional `background' hypothesis $\emptyset$.

PE\textsc{a}RL defines an energy function in terms of a set of labelling variables $f = \{f_p\}$, which specifies an index into $\mathcal{H}$ for each landmark $\vt{p}$, along with the set of plane parameters $\theta = \{\theta_h\}$
\begin{equation}
\label{eqn:pearl_energy}
E(f , \theta) = \sum_{\vt{p} \in \mathcal{L}} D_p( f_p, \theta_{f_p}) + 
\sum_{\vt{p},\vt{q} \in \mathcal{N}} V_{pq}(f_p, f_q) + 
\sum_{h \in \mathcal{H}} c_h \delta_h(f).
\end{equation}
The first term in this energy is a data cost, which specifies the cost for assigning each landmark to a given plane. For this we use 
\begin{equation}
D_p(f_p, \theta_{f_p}) = \left\{
\begin{aligned}
&\|\vt{p} - \theta_{f_p}\| && \text{if $f_p \neq \emptyset$} \\
&d_\emptyset && \text{otherwise}
\end{aligned}
\right.
\end{equation}
where $\|\vt{p} - \theta_{f_p}\|$ is the perpendicular distance between the landmark $\vt{p}$ and the plane with parameters $\theta_{f_p}$. For the background hypothesis $\emptyset$, a constant cost $d_\emptyset$ is used. In all our experiments we fix $d_\emptyset = 0.3$.

The second term in \eqref{eqn:pearl_energy} is a smoothness cost, which encourages neighbouring landmarks defined by a neighbourhood $\mathcal{N}$ to take the same label. In our case, we define a neighbourhood using the edges of the Delaunay triangulation which we originally computed for generating $\mathcal{H}$ and then use
\begin{equation}
V_{pq}(f_p, f_q) =  w_{pq}\mathbb{I}(f_p \neq f_q),
\end{equation}
where $\mathbb{I}$ is an indicator function, and
\begin{equation}
w_{pq} = \beta \, \text{exp}\left(-\frac{\|\vt{u}_p - \vt{u}_q\|^2}{\sigma_w^2}\right).
\end{equation}
Here $\vt{u}_p$ is the 2D position of landmark $\vt{p}$ when projected into the reference image (and likewise for $\vt{u}_q$), meaning that $w_{pq}$ is larger for pairs of landmarks which are closer together in the reference image. $\sigma_w$ is computed based on the mean value within the reference image $\sigma_w^2 = \frac{2}{|\mathcal{N}|}\sum_{p,q \in \mathcal{N}}\|\vt{u}_p - \vt{u}_q\|^2$ \cite{Boykov2001a}, and in all our experiments we fix the parameter $\beta = 0.05$. 

The final term in \eqref{eqn:pearl_energy} is a label cost, which plays the important role of controlling the number of planes which are active. Here\begin{equation}
\delta_h(f) = \left\{
\begin{aligned}
&1 && \text{if $\exists p : f_p = h$} \\
&0 && \text{otherwise},
\end{aligned}
\right.
\end{equation}
meaning every plane with non-zero support will incur a cost. The parameter $c_h$ controls how much cost is paid for each active hypothesis, and therefore by setting this parameter appropriately we can encourage solutions using a small number of planes, which is what we desire for our application. In our experiments we fix $c_h = 4$, except for the background hypothesis for which we use $c_\emptyset = 0$, as it should always be active and not penalised.

PE\textsc{a}RL then proceeds to minimise \eqref{eqn:pearl_energy} in an EM fashion: it first fixes plane parameters $\theta$ and optimises over the landmark labelling $f$, and then fixes $f$ and optimises over $\theta$. Since only the first term in \eqref{eqn:pearl_energy} is affected by $\theta$, for a given labelling $f$ we can simply perform a least-squares fit for each active plane independently, which is guaranteed to improve the solution. The original PE\textsc{a}RL algorithm \cite{Isack2011} proposed a heuristic means of optimising the labelling $f$ given a fixed $\theta$, but this was subsequently improved by Delong \etal \cite{Delong2011}, who proposed an extension to the $\alpha$-expansion \cite{Boykov2001} algorithm capable of handling the label cost term in \eqref{eqn:pearl_energy}, which is the method we use. These two optimisation steps constitute a single iteration of PE\textsc{a}RL, and in practice we find it sufficient to perform 3 iterations, as the method converges quickly.

At the end of this process, we are left with a set of active planes $\mathcal{P} = \{\pi_1, \dots, \pi_{N_P}\}$, along with an assignment of each landmark to one of these planes, or to the background class. An example assignment can be seen in Figure \ref{fig:reconstructor_planes}, in which the landmarks have been coloured to reflect their assignments to planes.

% maybe talk about how long this takes on a typical scene?

\section{Boundary estimation}
\label{sec:reconstructor_boundary_est}

The set of planes $\mathcal{P}$ gives us some information about the geometry of the scene, but because these planes have infinite extent they are of limited use in practice, since any augmentation would not respect physical boundaries of these planes in the scene. The next stage of our approach therefore aims to estimate boundary information for each of these planes. While we can already derive some information about plane extents by considering which landmarks have been assigned to each plane, this information is very sparse and is not sufficient to obtain a full reconstruction of the scene. Furthermore, by definition landmarks correspond to highly-textured points in the scene, meaning they do not generally provide information about regions with low texture. 

The approach we propose for estimating plane boundaries is to treat the task as a structured prediction problem and perform pixel-wise labelling of a reference view of the scene. Our goal is to assign each pixel in this view to one of the planes in $\mathcal{P}$, or to the background if it does not lie on any of these planes. Given such a labelling, we can then back-project the labels in the reference view onto each plane in order to obtain their extents. This approach is viewpoint-dependent and is therefore only able to produce a 2.5D reconstruction of the scene. However, this formulation considerably simplifies the resulting labelling problem, since in a particular view each pixel can only be assigned to a single plane, and we can therefore avoid explicitly handling the complex dependencies between planes such as how they occlude one another. This type of 2.5D reconstruction is also how most other approaches for real-time multi-view reconstruction proceed \cite{Newcombe2010,Newcombe2011,Stuehmer2010}.
% would be good to have a figure here

We begin by selecting a reference view of the scene, which is taken from the set of keyframes $\mathcal{K}$. Given that landmarks provide us with useful information for performing labelling, we choose the keyframe $K_r$ containing the largest number of landmark measurements to be our reference view. In order to keep computational requirements as low as possible, we first reduce the size of the labelling problem by over-segmenting the reference image $I_r$ to produce a set of superpixels $\mathcal{S}$. The method we choose for generating superpixels is the SLIC \cite{Achanta2010} algorithm, which is particularly computationally efficient, whilst producing regular superpixels that respect image boundaries well. This step reduces the size of the labelling problem dramatically from $320\times240 = 76,800$ pixels to roughly $1000$ superpixels. Example superpixel segmentations can be seen in Figure \ref{fig:reconstructor_slic}.

\begin{figure}[tb]
\centering
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=\textwidth]{images/slic1.png}
\end{subfigure}
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=\textwidth]{images/slic2.png}
\end{subfigure}
\caption[Examples of SLIC superpixels]{Examples of SLIC superpixels \cite{Achanta2010} for two reference images.}
\label{fig:reconstructor_slic}
\end{figure}

To find a labelling $L$ of the superpixels we define a pairwise CRF over the graph $\mathcal{G} = (\mathcal{S}, \mathcal{N})$, where $\mathcal{N}$ is the neighbourhood defined by pairs of superpixels which share a boundary. In doing so, we introduce structure into the resulting labelling problem, since we are making the assumption that the labels of neighbouring superpixels should affect one another. A good labelling then corresponds to the minimum of the energy function
\begin{equation}
\label{eqn:sp_energy}
E(L) = \sum_{s \in \mathcal{S}} D_s(L_s) + \sum_{s,t \in \mathcal{N}} V_{st}(L_s, L_t).
\end{equation}
We define the data term $D_s$ by considering the multi-view photo-consistency of pixels belonging to superpixel $s$. For each pixel $\vt{u}$ in $I_r$, given a particular plane $\pi$ we can calculate a hypothesised 3D position $\vt{X}^\pi_{\vt{u}}$ by back-projecting the pixel onto the plane. We select a small number of other keyframes from nearby viewpoints\footnote{In practice we choose the keyframes with the highest number of shared measurements with the reference keyframe, since this is a good indication that the viewpoints are nearby.} $\mathcal{K}^\rho = \{K^\rho_1, \dots, K^\rho_{N_\rho}\}$ to use for measuring photo-consistency, where in practice we take $N_\rho = 4$. The per-pixel photo-consistency cost for the plane $\pi$ is then defined as the average of the $L_1$-norm of the colour difference measured in Lab colour space in each keyframe\footnote{The average is only taken over those keyframes for which $\vt{X}^\pi_{\vt{u}}$ projects inside the image $I_o$.}
\begin{equation}
\label{eqn:reconstructor_photoconsistency}
\rho_{\vt{u}}(\pi) = \frac{1}{N_\rho} \sum_{K_o \in \mathcal{K}^\rho} \|I_r^{Lab}(\vt{u}) - I_o^{Lab}(\text{proj}(T_o^{-1}\vt{X}^\pi_{\vt{u}}))\|_1,
\end{equation}
where $\text{proj}(\cdot)$ projects a point from 3D camera space to 2D screen space. The $L_1$-norm is used in order to provide robustness against the situation where a pixel visible in the reference frame is occluded in another view, since we do not explicitly attempt to model these occlusions \cite{Newcombe2011}. In order to add some tolerance for sensor noise and slight inaccuracy in the estimates of camera poses, we first apply a Gaussian blur with $\sigma = 1.0$ to all images before computing this cost. The data term $D_s$ for a superpixel is then defined by taking the median cost for all pixels belonging to this superpixel, which provides additional tolerance to error caused by our relatively simple photo-consistency cost
\begin{equation}\label{eqn:reconstructor_photoconsistency_unary}
D_s(L_s) = \left\{
\begin{aligned}
&\underset{\vt{u} \in s}{\text{median}}(\rho_{\vt{u}}(L_s)) && \text{if $L_s \in \{\pi_1, \dots, \pi_{N_P}\}$} \\
&\rho_\emptyset && \text{otherwise}.
\end{aligned}
\right.
\end{equation}
% more on background label, since this is important
The background label $\emptyset$ presents a problem for this photo-consistency measure, since it is not possible to project a pixel into other views when it is assigned to the background, as the depth is unknown. The only option is therefore to use a constant cost in this case, which should be lower than the photo-consistency cost for typical incorrect plane assignments. This issue will be discussed more in Section \ref{sec:reconstructor_results}, but for all our experiments we have empirically chosen the value $\rho_\emptyset = 5$. Example photo-consistency costs for the example scene in Figure \ref{fig:reconstructor_example_scene} can be seen in Figure \ref{fig:reconstructor_photoconsistency_cost}.

The pairwise smoothness term $V_{st}$ in \eqref{eqn:sp_energy} should encourage smooth labellings of the reference image and is defined in terms of a combination of colour similarity between neighbouring superpixels and 3D depth information obtained from the plane geometry, similar to the approach taken by Gallup \etal \cite{Gallup2010}
\begin{equation}
V_{st}(L_s, L_t) = \beta \ V^c(s,t) \ V_{st}^d(L_s, L_t).
\end{equation}
Here $\beta$ is a constant scaling factor to ensure that the data and smoothness terms are comparable, in our experiments we fix $\beta = 15$. The first term is influenced by colour similarity and is defined as
\begin{equation}
V^c(s, t) = 0.2 + 0.8\ \text{exp}\left(-\frac{\|\bar{I_r}^{Lab}(s) - \bar{I_r}^{Lab}(t)\|^2}{\sigma_c^2}\right),
\end{equation}
where $\bar{I_r}^{Lab}(s)$ is the mean Lab colour over pixels $\vt{u} \in s$, and $\sigma_c^2 = \frac{2}{|\mathcal{N}|}\sum_{s,t \in \mathcal{N}}\|\bar{I_r}^{Lab}(s) - \bar{I_r}^{Lab}(t)\|^2$ \cite{Boykov2001a}. The effect of this term is to encourage transitions between labels to take place at colour discontinuities in the reference image, since these often correspond to the boundaries of planes. The second term is influenced by depth information between planes and defined as
\begin{equation}
V_{st}^d(L_s, L_t) = \left\{
\begin{aligned}
&0 && \text{if $L_s = L_t$} \\
&1 && \text{if $L_s = \emptyset$ or $L_t = \emptyset$} \\
&0.3 + 0.7 \ \text{min}(1, d/100) && \text{otherwise},
\end{aligned}
\right.
\end{equation}
where $d$ is the 3D depth difference between the centre of superpixels $s$ and $t$ according to their labels. The effect of this term is to encourage transitions between labels to take place at locations in the reference image corresponding to the projections of plane intersections, which should help to ensure the labelling respects the planes which have been identified.

Finally, we make use of the labelling of landmarks provided by the initial plane-fitting stage of our approach in order to provide hard constraints to guide the superpixel labelling $L$. This step is important as it allows us to inject the sparse labelling information which has already been obtained for the 3D landmarks into the resulting dense reconstruction. Each 2D landmark measurement $\vt{u}_m \in \mathcal{M}_r$ in the reference frame corresponds to a 3D landmark $\vt{p}_m$ which has now been assigned to a particular plane $\pi$ (or the background $\emptyset$). We therefore find the superpixel $s$ which contains $\vt{u}_m$ and modify $D_s$ such that it becomes
\begin{equation}
D_s(L_s) = \left\{
\begin{aligned}
&0 && \text{if $L_s = \pi$} \\
&\infty && \text{otherwise}.
\end{aligned}
\right.
\end{equation}
In cases where multiple measurements fall within the same superpixel but belong to different planes, we leave $D_s$ unchanged.
% mention how these are further filtered by looking for nearest neighbours?

The resulting energy \eqref{eqn:sp_energy} defines a standard multi-label pairwise CRF, for which an approximate solution can be efficiently found using the $\alpha$-expansion algorithm \cite{Boykov2001}.


\section{Online learning of plane appearance}
\label{sec:reconstructor_online_learning}

Multi-view photo-consistency provides a strong cue for performing the labelling of the reference view, however there are still situations where it is not informative or reliable. This is particularly the case for textureless regions, where the photo-consistency cost \eqref{eqn:reconstructor_photoconsistency} is generally unable to identify the correct plane assignment. Another issue is that the photo-consistency measure we use is computed in a rather simple manner to keep computational cost low and is therefore not very tolerant to slight inaccuracy in the keyframe poses estimated by the SLAM system, as well as different exposure settings or sensor noise in the camera images.

%Rather than introducing more sophisticated photoconsistency measures, we instead take the approach of learning appearance models

In order to handle these issues, we propose incorporating appearance information for each plane into the reconstruction pipeline. This is achieved by learning appearance models for each plane, which are initialised using the photo-consistency solution and subsequently used to refine the labelling in an iterative fashion. This approach is inspired by similar ideas which have been applied to interactive image segmentation, such as the GrabCut algorithm \cite{Rother2004}. 

% Assumption: planes have a consistent appearance which we can capture. The hope is that we capture some orthogonal information to what we get from photoconsistency alone, meaning the overall segmentation will improve

The approach we propose is to introduce a classifier which can be used to predict which plane a given superpixel belongs to, based on appearance information alone. For this purpose we use a multi-class linear SVM classifier learned in a one-vs-all manner \cite{Duan2005}, since this allows us to take advantage of efficient online SVM learning approaches \cite{Shalev-Shwartz2010}. For each superpixel $s$, we construct a feature vector $\vt{x}_s$ defined by the bins of a 3D colour histogram. This histogram uses 5 bins per colour channel, meaning $\vt{x}_s$ is a 125D vector with $\|\vt{x}_s\|_1 = 1$. For each plane $\pi$, as well as the background class $\emptyset$, we introduce a linear weight vector $\vt{w}_\pi$. Given a labelling $L$, we can generate positive and negative training examples $\mathcal{X}_\pi^+ = \{\vt{x}_s \ | \ L_s = \pi\}$ and $\mathcal{X}_\pi^- = \{\vt{x}_s \ | \ L_s \neq \pi\}$ for each plane, which are then used to update the associated weight vectors by performing online learning using the Pegasos algorithm \cite{Shalev-Shwartz2010} (which was also described in Section \ref{sec:background_online_learning} of this thesis). For all our experiments, we use an SVM regularisation of $C=0.1$.

In order to use such a classifier to produce a unary cost for the labelling energy function, we take the approach of Kumar and Hebert \cite{Kumar2006} and use a logistic function to produce a per-plane likelihood for each superpixel from the SVM classification score:
\begin{equation}
P(\vt{x}_s | L_s) = \frac{1}{1 + \exp(- \langle \vt{w}_{L_s}, \vt{x}_s \rangle)}.
\end{equation}
This likelihood will be close to 1 when the classification score is large and positive and close to 0 when it is large and negative. We then take the negative log-likelihood to be the unary superpixel cost:
\begin{equation}\label{eqn:reconstructor_appearance_unary}
D_s^{app}(L_s) = -\log P(\vt{x}_s | L_s) = \log(1 + \exp(- \langle \vt{w}_{L_s}, \vt{x}_s \rangle))
\end{equation}
This appearance cost is combined with the original photo-consistency cost to give
\begin{equation}
\label{eqn:reconstructor_combined_unary}
D_s(L_s) = \frac{1}{2}(D_s^{pc}(L_s) + \gamma D_s^{app}(L_s))
\end{equation}
where $D_s^{pc}$ is the original cost defined in Section \ref{sec:reconstructor_boundary_est}, and $\gamma$ is a parameter to ensure the scales of the two costs are comparable. In all our experiments we fix $\gamma = 10$. Appearance costs produced after training the SVM classifier from the initial photo-consistency labelling for the example scene in Figure \ref{fig:reconstructor_example_scene} can be seen in Figure \ref{fig:reconstructor_svm_cost}.

\begin{sidewaysfigure}[p]
\centering
\begin{subfigure}[b]{\textwidth}
\includegraphics[width=\textwidth]{images/photoconsistency_cost.png}
\caption{Photo-consistency cost}
\label{fig:reconstructor_photoconsistency_cost}
\end{subfigure}

\vspace{1cm}

\begin{subfigure}[b]{\textwidth}
\includegraphics[width=\textwidth]{images/svm_cost.png}
\caption{Appearance cost}
\label{fig:reconstructor_svm_cost}
\end{subfigure}

\caption[Example superpixel unary costs]{Example superpixel unary costs for the tabletop scene in Figure \ref{fig:reconstructor_example_scene}. (a) shows the photo-consistency cost computed according to \eqref{eqn:reconstructor_photoconsistency_unary} for each of the 4 planes identified in Figure \ref{fig:reconstructor_planes}. (b) shows the appearance cost computed according to \eqref{eqn:reconstructor_appearance_unary} using the SVM classifier trained from the initial photo-consistency solution. Note that in (b) the left-most cost is for the background label, for which it is not possible to compute a photo-consistency cost.}
\end{sidewaysfigure}


Our overall approach proceeds as follows: we first find an initial labelling $L^0$ using photo-consistency information alone (as described in Section \ref{sec:reconstructor_boundary_est}). We then use this labelling in order to learn the weight vectors $\{\vt{w}_\pi^0\}$ for each plane. These weight vectors are subsequently used to define the combined data cost \eqref{eqn:reconstructor_combined_unary}, resulting in a new labelling $L^1$. This new labelling is used to update the weight vectors per plane $\{\vt{w}_\pi^1\}$, and the process is repeated for a number of iterations, in a similar manner to GrabCut \cite{Rother2004}.

In common with the other approaches which have been presented in this thesis, we are therefore making use of online learning in order to provide an element of adaptability to a given environment with this approach. Our motivation is that the use of photo-consistency information provides a good starting point for a scene reconstruction and will succeed in many areas. However, there are other areas which will not be well reconstructed using photo-consistency information alone, and the hope is that plane appearance provides an orthogonal cue which will allow information to be transferred to the uncertain regions, resulting in a more consistent overall reconstruction.

Unlike the structured learning approaches presented in Chapters \ref{chp:struck} and \ref{chp:learnmatch}, in this approach the classifier does not explicitly take structure into account, as it is trained to classify superpixels independently. However, it is worth noting that the way in which this classifier is trained \emph{does} take the structure into account. The samples used to train the classifier are generated based on the final CRF labelling, which has been found using the pairwise neighbourhood structure of the superpixels in the reference image. Thus our approach can still be seen as performing a form of structured learning, with the structure being taken into account implicitly by the learning procedure.



% talk about how this can be seen as a form of structured learning, since the labels are provided by the output of labelling, which has already taken structure into account.


% figures?

% multiview consistency provides a strong cue for assigning superpixels to planes. however, there are a couple of issues: ambiguous regions (e.g. textureless arease), and inaccuracy or noise in the consistency maps (especially since we perform a cheap matching). consequently, it is difficult to obtain reliable reconstructions of the scene.

% one way to tackle this would be to come up with more sophisticated ways of doing the multi view consistency, e.g. better matching measures, taking into account uncertainty, using more keyframes etc. however all of these add significant expense, and we want an approach for low-powered devices.

% instead, we propose to try and learn about the appearance of each plane, and use this as additional information when performing the segmentation. our hypothesis is that we can improve upon the initial multiview labelling to guide an online learner, and then we can use some of the techniques described in previous chapters.

% we use features from the superpixels based on colour, and combine this with simple learning techniques.

% *** explain that we are learning the unary potential within a structured prediction framework. because we are using the structured prediction to supply our labelling for self-training, this can be seen as a limited form of structured learning. 

% explain how we iterate by first using multiview only, and then add in the appearance cost. give an overall algorithm box for our entire method


\chapter{Results}
\label{sec:reconstructor_results}

% example output from the approach

\begin{figure}[htbp]
\centering

\begin{subfigure}[c]{0.155\textwidth}\centering
\includegraphics[width=\textwidth]{images/multi_ref_image.png}
\end{subfigure}
\begin{subfigure}[c]{0.2\textwidth}\centering
\includegraphics[width=\textwidth]{images/multi_label1.png}
\end{subfigure}
\begin{subfigure}[c]{0.2\textwidth}\centering
\includegraphics[width=\textwidth]{images/multi_label2.png}
\end{subfigure}
\begin{subfigure}[c]{0.2\textwidth}\centering
\includegraphics[width=\textwidth]{images/multi_label3.png}
\end{subfigure}
\begin{subfigure}[c]{0.2\textwidth}\centering
\includegraphics[width=\textwidth]{images/multi_label4.png}
\end{subfigure}

\begin{subfigure}[c]{0.155\textwidth}\centering
\includegraphics[width=\textwidth]{images/eyepet2_ref_image.png}
\end{subfigure}
\begin{subfigure}[c]{0.2\textwidth}\centering
\includegraphics[width=\textwidth]{images/eyepet2_label1.png}
\end{subfigure}
\begin{subfigure}[c]{0.2\textwidth}\centering
\includegraphics[width=\textwidth]{images/eyepet2_label2.png}
\end{subfigure}
\begin{subfigure}[c]{0.2\textwidth}\centering
\includegraphics[width=\textwidth]{images/eyepet2_label3.png}
\end{subfigure}
\begin{subfigure}[c]{0.2\textwidth}\centering
\includegraphics[width=\textwidth]{images/eyepet2_label4.png}
\end{subfigure}

\begin{subfigure}[c]{0.155\textwidth}\centering
\includegraphics[width=\textwidth]{images/books2_ref_image.png}
\end{subfigure}
\begin{subfigure}[c]{0.2\textwidth}\centering
\includegraphics[width=\textwidth]{images/books2_label1.png}
\end{subfigure}
\begin{subfigure}[c]{0.2\textwidth}\centering
\includegraphics[width=\textwidth]{images/books2_label2.png}
\end{subfigure}
\begin{subfigure}[c]{0.2\textwidth}\centering
\includegraphics[width=\textwidth]{images/books2_label3.png}
\end{subfigure}
\begin{subfigure}[c]{0.2\textwidth}\centering
\includegraphics[width=\textwidth]{images/books2_label4.png}
\end{subfigure}

\begin{subfigure}[c]{0.155\textwidth}\centering
\includegraphics[width=\textwidth]{images/movebox_ref_image.png}
\end{subfigure}
\begin{subfigure}[c]{0.2\textwidth}\centering
\includegraphics[width=\textwidth]{images/movebox_label1.png}
\end{subfigure}
\begin{subfigure}[c]{0.2\textwidth}\centering
\includegraphics[width=\textwidth]{images/movebox_label2.png}
\end{subfigure}
\begin{subfigure}[c]{0.2\textwidth}\centering
\includegraphics[width=\textwidth]{images/movebox_label3.png}
\end{subfigure}
\begin{subfigure}[c]{0.2\textwidth}\centering
\includegraphics[width=\textwidth]{images/movebox_label4.png}
\end{subfigure}

\begin{subfigure}[c]{0.155\textwidth}\centering
\includegraphics[width=\textwidth]{images/pompom_ref_image.png}
\end{subfigure}
\begin{subfigure}[c]{0.2\textwidth}\centering
\includegraphics[width=\textwidth]{images/pompom_label1.png}
\end{subfigure}
\begin{subfigure}[c]{0.2\textwidth}\centering
\includegraphics[width=\textwidth]{images/pompom_label2.png}
\end{subfigure}
\begin{subfigure}[c]{0.2\textwidth}\centering
\includegraphics[width=\textwidth]{images/pompom_label3.png}
\end{subfigure}
\begin{subfigure}[c]{0.2\textwidth}\centering
\includegraphics[width=\textwidth]{images/pompom_label4.png}
\end{subfigure}

\begin{subfigure}[c]{0.155\textwidth}\centering
\includegraphics[width=\textwidth]{images/monitors_ref_image.png}
\end{subfigure}
\begin{subfigure}[c]{0.2\textwidth}\centering
\includegraphics[width=\textwidth]{images/monitors_label1.png}
\end{subfigure}
\begin{subfigure}[c]{0.2\textwidth}\centering
\includegraphics[width=\textwidth]{images/monitors_label2.png}
\end{subfigure}
\begin{subfigure}[c]{0.2\textwidth}\centering
\includegraphics[width=\textwidth]{images/monitors_label3.png}
\end{subfigure}
\begin{subfigure}[c]{0.2\textwidth}\centering
\includegraphics[width=\textwidth]{images/monitors_label4.png}
\end{subfigure}

\begin{subfigure}[c]{0.155\textwidth}\centering
\includegraphics[width=\textwidth]{images/cluttered2_ref_image.png}
\end{subfigure}
\begin{subfigure}[c]{0.2\textwidth}\centering
\includegraphics[width=\textwidth]{images/cluttered2_label1.png}
\end{subfigure}
\begin{subfigure}[c]{0.2\textwidth}\centering
\includegraphics[width=\textwidth]{images/cluttered2_label2.png}
\end{subfigure}
\begin{subfigure}[c]{0.2\textwidth}\centering
\includegraphics[width=\textwidth]{images/cluttered2_label3.png}
\end{subfigure}
\begin{subfigure}[c]{0.2\textwidth}\centering
\includegraphics[width=\textwidth]{images/cluttered2_label4.png}
\end{subfigure}

\begin{subfigure}[c]{0.155\textwidth}\centering
\includegraphics[width=\textwidth]{images/0002_ref_image.png}
\end{subfigure}
\begin{subfigure}[c]{0.2\textwidth}\centering
\includegraphics[width=\textwidth]{images/0002_label1.png}
\end{subfigure}
\begin{subfigure}[c]{0.2\textwidth}\centering
\includegraphics[width=\textwidth]{images/0002_label2.png}
\end{subfigure}
\begin{subfigure}[c]{0.2\textwidth}\centering
\includegraphics[width=\textwidth]{images/0002_label3.png}
\end{subfigure}
\begin{subfigure}[c]{0.2\textwidth}\centering
\includegraphics[width=\textwidth]{images/0002_label4.png}
\end{subfigure}

\begin{subfigure}[b]{0.155\textwidth}\centering
\caption{}
\end{subfigure}
\begin{subfigure}[b]{0.2\textwidth}\centering
\caption{}
\end{subfigure}
\begin{subfigure}[b]{0.2\textwidth}\centering
\caption{}
\end{subfigure}
\begin{subfigure}[b]{0.2\textwidth}\centering
\caption{}
\end{subfigure}
\begin{subfigure}[b]{0.2\textwidth}\centering
\caption{}
\end{subfigure}

\caption[Result of the proposed method on tabletop scenes]{Result of the proposed method on a number of tabletop scenes. The reference image is shown in column (a), and column (b) shows the result of labelling using photo-consistency cost alone. Columns (c)-(e) show the result of labelling after each iteration of updating the appearance-based classifier. In all cases the dark blue colour corresponds to the background label.}
\label{fig:reconstructor_examples}
\end{figure}

% show some good results, and some failure cases

Typical results produced by our approach on a number of example desktop scenes can be seen in Figure \ref{fig:reconstructor_examples}. For each scene this figure shows the reference image, along with the initial result found using photo-consistency information alone (Section \ref{sec:reconstructor_boundary_est}). Subsequent columns show the result after each iteration of re-labelling using the appearance-based classifier (Section \ref{sec:reconstructor_online_learning}), which shows how the resulting labelling changes as appearance information is incorporated and updated.

As can be seen from these results, in many cases the proposed method is able to produce promising coarse scene reconstructions. For the applications we are interested in, namely providing basic scene reconstruction for gaming applications, these reconstructions would often be adequate. In most cases, the actual boundaries would not need to be displayed to the user, but rather they would be used internally by a game in order to allow virtual content to respect physical boundaries in the scene. The boundaries identified by our approach would be sufficient for defining collision geometry for a physics engine, or for performing occlusion of virtual objects as they move behind objects in the scene.

% discuss how the online learning stage helps

We see from these results that the plane-finding stage of our approach (Section \ref{sec:reconstructor_plane_finding}) is rather robust and able to reliably find a small number of dominant planes in the scene using the landmarks from the SLAM map. We have found empirically that this stage is not very sensitive to parameter settings and also requires minimal computational cost.

The boundary estimation stage of our approach is less robust, however, and we see from the results that it does not reliably produce high-quality segmentations. The primary issue is that the method aims to be fully automatic, and as a result the iterative online learning algorithm we use can fail to refine the solution if the initial labelling provided by the photo-consistency cost contains gross errors. Essentially, our method is performing an unsupervised clustering of the scene, and if the cluster initialisations are poor, the final labelling will also suffer. Another issue is that there are a relatively large number of parameters involved in defining the energy function which is minimised when performing labelling, and although these have all been fixed throughout our experiments, they have been set empirically by hand.

Perhaps the most serious difficulty with this approach is the requirement of having a background label. As has previously been mentioned, because pixels labelled as background have unknown depth, we must use a fixed value for the photo-consistency cost for this label. Choosing this value to work in all cases is difficult: if it is too high then regions which should be labelled as background are assigned to planes, which subsequently is reinforced when learning the appearance models for planes; conversely, if it is too low then regions which should be assigned to planes are assigned to the background, which also is reinforced once the appearance models are learned. 

% show a figure illustrating the problems with background label (different values of background cost)

% quantatitive evaluation on labeled data ???
% leave this until the end, it can be done if totally required, but otherwise forget it

\section{Implementation on a low-powered device}

The work in this chapter was originally motivated by the desire to perform scene reconstruction on a low-powered device. To demonstrate that the approach we have developed is indeed suitable for this setting, we have produced an implementation for the PlayStation Vita portable games console.

After an initial period of exploring the scene in order to build up a map, we then trigger our reconstruction algorithm. The reconstruction algorithm only needs to be run once in order to produce 3D geometry which can subsequently be used by AR applications, so it is sufficient for it to be able to execute within a few seconds, which could then be run as a background task as part of a game. Table \ref{tbl:reconstructor_vita_timings} shows timings for our method when running on this hardware. In this implementation, we first perform labelling using photo-consistency information alone and then perform three iterations of appearance learning and re-labelling to produce the final reconstruction. The computational complexity of some of the stages involved are affected by the number of active planes, so we show timings for scenes in which 1-5 planes have been identified. In all cases the timings are produced by averaging the results over 3 different scenes with the given number of planes.

% need to check all of these
\begin{table}[htb]
\centering
\begin{tabular}{|l|r|r|r|r|r|}
\hline
Stage & 1 plane & 2 planes & 3 planes & 4 planes & 5 panes \\
\hline

Plane finding & 62 & 62 & 62 & 62 & 62\\
Blurring & 59 & 59 & 59 & 59 & 59\\
Photo-consistency & 810 & 1388 & 1971 & 2557 & 3150\\
SLIC & 391 & 391 & 391 & 391 & 391\\
\hline
Labelling & 155 & 188 & 205 & 227 & 254\\
Learner update & 5 & 10 & 14 & 19 & 24\\

\hline

{\bf Total} & {\bf 1957} & {\bf 2687} & {\bf 3354} & {\bf 4048} & {\bf 4769}\\

\hline

\end{tabular}
\caption[Timings of our approach running on a PlayStation Vita]{Timings (ms) of our approach running on a PlayStation Vita. We first perform labelling using photo-consistency information alone, followed by 3 iterations of appearance learning and re-labelling to produce the final reconstruction.}
\label{tbl:reconstructor_vita_timings}
\end{table}

We see from these results that our approach is able to produce results for up to 5 planes within approximately 5 seconds. It should be noted that these timings are for an unoptimised C++ implementation, and we therefore would expect that all of these could be significantly improved with a more careful implementation. Nevertheless, the fact that the approach can run with a few seconds would be acceptable for a background task during a game. The most expensive operation is currently the building of the per-pixel photo-consistency cost, since this involves warping the keyframe images with a homography defined by each of the planes which have been identified. As the number of planes increases, this stage dominates the overall time taken by the algorithm. However, we anticipate that this stage in particular could be significantly optimised, which would therefore make the overall algorithm much faster.

\begin{figure}[p]
\centering
\includegraphics[width=\textwidth]{images/vita_deskscene.jpg} \\
\vspace{0.1cm}
\includegraphics[width=\textwidth]{images/vita_computerscene.jpg}
\caption[Our method running on a PlayStation Vita]{Examples results of the proposed reconstruction algorithm running on a PlayStation Vita portable games console. Here the labelling has been back-projected to produce a 3D mesh for each plane defined by the centres of superpixels.}
\label{fig:reconstructor_vita}
\end{figure}

Figure \ref{fig:reconstructor_vita} shows some examples of the reconstruction system running on this hardware, where the results of labelling have been back-projected to produce a 3D mesh defined by the centres of the superpixels. Although coarse, we can see that this geometry would be sufficient for the purpose of adding AR content realistically into the scene.



% these figures suck

\chapter{Summary}
\label{sec:reconstructor_conclusions}

In this chapter we have presented a method for performing coarse 3D reconstruction of tabletop scenes intended for AR applications based around SLAM. Our motivation was to produce an approach suitable for use on low-powered devices, and we have shown that this has been achieved with an implementation which operates within a few seconds on a PlayStation Vita portable games console.

Our approach makes use of simple photo-consistency information to obtain an initial reconstruction and subsequently uses online SVM learning of plane appearance in order to produce a more refined solution. We have demonstrated how this use of online learning allows us to transfer information from regions which are well-reconstructed using photo-consistency information to those which are not, such as textureless regions, resulting in a more complete overall reconstruction.

While the framework we have presented shows promising results, there are still a number of outstanding issues and avenues for future research which we believe would make for a more robust and practically useful solution.

The first issue is how best to handle the background class, which is required for labelling regions of the reference image which do not belong to any plane. This class requires a constant photo-consistency cost to be used and choosing this value to work across all scenes is difficult in our current framework. One approach for tackling this issue could be to make this value adaptive and attempt to estimate it online for a given scene in order to give a stable reconstruction.

Another avenue for future work would be to improve the accuracy of the labelling of the reference view. One issue at present is that our algorithm contains a relatively large number of parameters which have been set by hand, so it would most likely be beneficial to try and learn these parameters based on labelled training data, which has been shown to be beneficial for other pixel-wise labelling problems \cite{Alahari2010,Szummer2008}. Accuracy could potentially also be improved by including additional features besides colour histograms when learning the per-plane appearance classifier. Texture, for example, or more sophisticated features \cite{Shotton2007,Shotton2008} could potentially provide stronger cues for classification.

Finally, the reconstruction produced by our algorithm is currently 2.5D, as we generate per-pixel depth information for the reference view. For a more complete 3D scene reconstruction, the reconstructions from multiple reference views could be fused together, in an approach similar to that used by other real-time reconstruction methods \cite{Newcombe2010,Newcombe2011}. 


% Our approach suitable for low-powered devices, and gives reasonable results. 

% Still a lot of room for improvement, however, and future work would include...
% not committing to planes early on
% improved accuracy of segmentation
% fusing of results from multiple reference frames
% learning parameters, many set by hand
