Model-Based Adaptive Predictive Control with Visual Servo of a Rotary Crane System

1.   Introduction

Many papers in the field of stereo vision^[1]-[6] have been published in the past decade. The progress has been made in improving accuracy as well as achieving real-time performance by using multiple cameras. However, using cameras to compute 3D position coordinate of targets is still challenging due to the unpredictable lighting conditions, unavoidable noise, and interferences of the surrounding objects.

In this paper, the proposed rotary crane system is equipped with a camera for payload monitoring. This arrangement is important and novel in building construction, factories, and harbors as it avoids the installation of durable encoders at the hanging base of string. To achieve the reliable position estimation, a white light emitting diode (LED) is attached on the payload and a camera is installed on the rotary frame to derive swing angles for control feedback. The model predictive control (MPC)^[7],[8] is applied on this system. Importantly, the motors in the system are controlled in velocity mode, which much simplifies the consideration of nonlinear effects caused by friction and load. Furthermore, to compensate for the time-delay effects in visual servo, a robust adaptive predictive controller is proposed by using echo state neural (ESN) network^[9],[10].

2.   3D Modeling by a Camera

The coordinate systems are shown in Fig. 1. The coordinate transformation of the coordinates attached on the rotary crane and the camera as in Fig. 1, is defined as follows:

Figure  1.  Coordinates of the proposed rotary crane: (a) picture of the crane system showing the coordinate systems and (b) schematic diagram of the coordinate systems.

DownLoad: Full-Size Img PowerPoint

For the camera coordinate system, as shown in Fig. 2, there are an image plane π and a camera frame center O. The distance between O and π is the focal length f, and the projective point p is the intersection of the line connecting point P and O with the image plane π. By the properties of similar triangles, the 2D image coordinate is calculated as follows:

Figure  2.  Coordinate of the camera.

DownLoad: Full-Size Img PowerPoint

In order to find the solutions of ${ {\textit{α}} ,\,\,{\textit{β}} , \,\, {\textit{γ}} ,\,\, d,\,\, h,\,\, l,\,\, p, {\rm and}\;f }$ of this 3D position coordinate model, the downhill simplex method is used with the experimental measurements served as training data, as listed in Table 1. The results are shown in Table 2. Next, considering that

Table  1.  Relationships between $({\textit{θ}} ,{\textit{ϕ}} )$ and $(x,y)$ using experimental measurements

	θ (degree)	θ (degree)
		20	30	40
x	180	–114.300	–133.630	–164.630
	135	–86.970	–93.630	–115.300
	90	–19.300	–19.800	–24.630
	45	66.200	70.370	86.700
	0	115.030	128.533	166.030
	–45	92.370	108.370	143.030
	–90	10.370	12.200	19.370
	–135	–70.133	–98.300	–113.630
y	180	13.070	17.070	29.233
	135	53.900	60.570	86.070
	90	74.570	81.400	108.070
	45	63.730	73.070	97.070
	0	29.400	32.833	54.570
	–45	–16.500	–15.767	–14.330
	–90	–38.600	–42.930	–49.600
	–135	–29.517	–30.767	–32.930

 | Show Table

DownLoad: CSV

Table  2.  Optimal device parameters of the rotary crane and the camera

d (mm)	h (mm)	l (mm)	p (mm)
558.78	351.04	302.82	25.78
β (degree)	γ (degree)	α (degree)	f
–6.15	18.43	–9.50	603.14

 | Show Table

DownLoad: CSV

Moreover, we notice that the position of the swinging load can be uniquely defined by ${\textit{α}}$ and ${\textit{β}}$, as shown in Fig. 3. Hence, we use the Kriging algorithm to find the 3D Kriging surface model for interpolating 3D coordinates, using ${\textit{α}}\,\,{\rm and} \,\,{\textit{β}}$ as input, for the purpose of minimizing the visual delay, as shown in Fig. 4.

Figure  3.  Definition of swinging angles.

DownLoad: Full-Size Img PowerPoint

Figure  4.  Model of position and swinging angles: (a) and (b) are the model between the position (${p_x}, {p_y}$) and the swinging angles α and β, respectively; (c) and (d) are the 3D Kriging surface fitting model for the position (${p_x}, {p_y}$) and the swinging angles α and β, respectively.

DownLoad: Full-Size Img PowerPoint

3.   Predictive Control

In this section, a general scheme of Fig. 5 is proposed in the predictive control to extend the control horizon for more than one step in order to compensate for the delayed sensor feedback using a camera^[7],[8],[11].

Figure  5.  Control structure of this study.

DownLoad: Full-Size Img PowerPoint

The physical model of the rotary crane system is described as follows:

where ${b_1}$ and ${b_2}$ are damping coefficients to be decided. Equations (1) and (2) are in the standard state space form:

where the state vector ${\bf x} = {[{\textit{θ}} ,\;\dot {\textit{θ}} ,\;{\textit{φ}} ,\;\dot {\textit{φ}} ]^T},$ the control input vector ${\bf u} = {[\dot {\textit{α}} ,\;\ddot {\textit{α}} ,\;\dot r,\;\ddot r]^T}$, and the output vector ${\bf y} = {[{{\textit{α}} _g},\;{{\textit{γ}} _g}]^T}$, as shown in the experimental platform of Fig. 6.

Figure  6.  Picture of the experimental prototype setup.

DownLoad: Full-Size Img PowerPoint

During the time interval between $t$ and $t + {T_C}$, the ESN-model-based controller ${u^*}$ simulates a physical model described by:

using estimated initial state vector $\hat {\bf x}(t + {T_C})$, where $t = k {T_s}$, ${T_s}$ is sampling period and $k = 0,1,2,\cdots$; ${T_P}$ is the prediction horizon, and ${{\bf y}_r}$ is the desired output of model output $\hat {\bf y}({\textit{τ}} )$. Relations between these variables and parameters are shown in Fig. 7.

Figure  7.  Relations between the reference trajectory, candidate control inputs, and estimated system output during the predictive control.

DownLoad: Full-Size Img PowerPoint

As shown in Fig. 7, $\hat {\bf x}(t + {T_C})$ is obtained by the simulation of (4) using the control input sequence calculated in the last time interval $[t,\;t - {T_C}]$. For each trial, taking ${\bf u}_i^*$ as the i-th trial, different weights ${\textit{ϖ}}$, ${\textit{ψ}}$ and ${\textit{ζ}}$ are applied in the two orthogonal directions, ${\textit{α}}$ and $r$, in different stages. In doing so, the cost function is defined as:

where ${{\textit{α}} _d}({\textit{τ}} )$ and ${{\textit{γ}} _d}({\textit{τ}} )$ are desired trajectories; ${\hat {\textit{α}} _g}({\textit{τ}} )$ and ${\hat {\textit{γ}} _g}({\textit{τ}} )$ are estimated trajectories of ${{\textit{α}} _g}$ and ${{\textit{γ}} _g}$, respectively .

4.   Real-Time Implementation

In this section, a tapped-delay^[12] ESN network^[13]-[15] is designed as an adaptive model for cubic spline interpolation^[16],[17] to enlarge the predictive horizon^[18],[19], where the physical model-based controller^[20] ${{\bf u}^*}$ in (4) is replaced by a discrete-time ESN mode composed of 10 neurons each equipped with 9 tapped-delay.

Furthermore, an adaptation theorem is used for modeling to enhance the accuracy of ESN-based model. First, the input sequence is assumed to be a $K \times 1$ matrix, ESN has N neurons and the output is an $L \times 1$ matrix, moreover, the state of each neuron is ${\bf x}(k) = {\bf \tanh} ({{\bf W}^{{\rm{in}}}} \cdot {\bf u}(k) + {\bf W} \cdot {\bf x}(k - 1) + {{\bf W}^{{\rm{back}}}})$, where the length of ${{\bf W}^{{\rm{in}}}}$ is an $N \times K$ matrix, ${\bf W}$ is $N \times N$ and the ${{\bf W}^{{\rm{back}}}}$ is $N \times L$. The final outputs of model is $\hat {\bf y}(k) ={{\bf W}^{{\rm{out}}}} \cdot$$[{\bf u}(k),{\bf x}(k),\hat {\bf y}(k - 1),\cdots\!,\;\;\hat {\bf y}(k - n)]$.

To complete the neural plant modeling design, a convergence theory for this model is provided in the following theorem:

Theorem 1. If the sampling time ${T_s}$ is selected such that $\left\| {\bar {\bf y}(k) - {\bf y}(k)} \right\| \leqslant$$\bar {\textit{ε}}$ and $0 < {\textit{η}} (k) < {2 / {{{\left\| {\frac{{{\rm{d}}\hat {\bf y}(k)}}{{{\rm{d}}{{\bf W}^{{\rm{out}}}}(k)}}} \right\|}^2}}}$, is satisfied, where

where $\bar {\bf y}(k)$ is the optimal output of model, we have that $\hat {\bf y}(k) \to \bar {\bf y}(k) \approx {\bf y}(k)$.

Proof of Theorem 1. First, the Lyapunov candidate^[21] is designed as follows,

where ${V_2}(k) = {1}/{2}{\left\| {\hat {\bf y}(k) - \bar {\bf y}(k)} \right\|^2}$, ${\textit{ε}} (k) = {1}/{2}{\left\| {\bar {\bf y}(k) - {\bf y}(k)} \right\|^2} +(\hat {\bf y}$$(k) -\bar {\bf y}(k))^T (\bar {\bf y}(k) - {\bf y}(k))$.

This model is then trained such that ${V_2}(k)$ is minimized. The following derivation involves a convergence condition for selecting appropriate learning rates ${\textit{η}} (k)$. Firstly we notice that

where ${\bf e}(k) = \hat {\bf y}(k) - \bar {\bf y}(k)$. Thus

Hence, if $- 1 < 1 - {\left[ {\displaystyle\frac{{{\rm{d}}\hat {\bf y}(k)}}{{{\rm{d}}{{\bf W}^{{\rm{out}}}}(k)}}} \right]^T}{\textit{η}} (k)\displaystyle\frac{{{\rm{d}}\hat {\bf y}(k)}}{{{\rm{d}}{{\bf W}^{{\rm{out}}}}(k)}} < 1$ and $\| \bar {\bf y}(k) -$${\bf y}(k) \| \leqslant \bar {\textit{ε}}$, we have that ${V_2}(k + 1) < {V_2}(k)$. This is equivalent to ${V_2}(k) \to 0$, thereby the proof is completed.

Input to the ESN model are past positions, velocities, and accelerations of motor and payload. Outputs of the ESN model are now positions of payload. Each trial control sequence is encoded into 3 parameters which define a cubic spline.

In order to find the best solution on-line, the downhill simplex method is used. Considering that the method suffers from the possibility of being trapped in local minima, we restart each interval with four polytopes of 4 vertices randomly and make it calculate for 20 iterations, and then start a polytope with the 4 best results of these polytopes as vertices and run for 20 more iterations.

5.   Experimental Verification

Experimental prototype of Fig. 6 was used for several verification experiments. In the experimental settings, the cable length varies between 0.175 m and 0.400 m, and the payload is assumed to be 2.851 kg hanging from a cable of 920 mm with nominal length being 310 mm, and the control signal is updated with a sampling period of 0.05 s, according to the experimental prototype, and ${T_P} = 10 {T_C}$. Longer ${T_C}$ alleviates the demand on computing power of the controller.

Firstly, the effect of disturbance is studied by assuming an extra velocity of 57.3 deg/s in the ${\textit{θ}}$ direction during the period of 5.5 s to 6.5 s. The best control horizon ${T_C} = 0.3$ s without inducing significant vibration is selected through testing the tracking performance of the system for control horizons ranging from 0.1 s to 0.6 s. Fig. 8 shows the responses when disturbance is applied. Finally, the advantage of adaptive predict control is dictated by comparing the results with the conventional predictive control^[11] in Fig. 9.

Figure  8.  Tracking performance of the predictive controller when an extra disturbance is introduced between 5.5 s to 6.5 s: (a) a_g and (b) y_g.

DownLoad: Full-Size Img PowerPoint

Figure  9.  Comparison and weights: (a) and (b) are comparison of swing ranges between the conventional predictive controller and the proposed adaptive predict controller, where (a) shows the movement of the payload in the left and right direction and (b) in the forward and backward direction, both are expressed in meters; (c) weights of ESN model are added up showing the trends of adaptation. The inputs to the ESN model are normalized that the weights have no units.

DownLoad: Full-Size Img PowerPoint

6.   Conclusions

The control problems of modeling error and time-varying delay in anti-swing control were handled with the adaptive predictive control strategy. Importantly, a robust LED vision servo was proposed, and a tapped-delay ESN network was designed as an adaptive model to enlarge prediction horizon in the control law. Performances of the proposed scheme were verified by experimental results.