Article - Control

Transactions of the Institute of

Measurement and Control
Nonlinear stabilization and reference 2024, Vol. 46(2) 301–315
Ó The Author(s) 2023

tracking of visual servo system using Article reuse guidelines:

sagepub.com/journals-permissions
DOI: 10.1177/01423312231169163
TS fuzzy augmented iterative learning journals.sagepub.com/home/tim

control: Experimental validation

Vimala Kumari Jonnalagadda1 and Vinodh Kumar Elumalai2

Abstract
To address the nonlinear stabilization problem and improve the tracking control feature of ball on plate system (BPS), this paper puts forward a
novel Takagi Sugeno (TS) fuzzy control augmented with the current cycle feedback iterative learning control (CCF-ILC) scheme. According to
Bode’s sensitivity integral, the performance of linear controllers is always a trade-off between reference tracking and robustness. Hence, to deal
with the so-called ‘waterbed’ effect, this work exploits the capability of TS fuzzy to handle the nonlinear dynamics and synthesizes a learning con-
trol scheme based on current iteration error to capitalize the information rich error signal for enhancing the robustness and trajectory tracking
features. The global asymptotic stability of the proposed TS fuzzy augmented ILC scheme is proved using the Lyapunov function and linear matrix
inequalities (LMIs). Moreover, the monotonic convergence of ILC is presented based on the singular value condition. For identifying the rolling
mass from the video stream, a background subtraction algorithm based on thresholding technique is implemented. Finally, the robustness and
tracking features of the proposed scheme are evaluated on a two degrees of freedom (DoF) laboratory scale BPS system through hardware in loop
(HIL) testing for three realistic test cases. The tracking performance quantified using the root mean square error (RMSE) and power spectral den-
sity plot corroborates that the proposed scheme can offer better setpoint tracking and robustness feature compared to state-of-the-art fuzzy and
ILC control techniques implemented on BPS.

Keywords
TS fuzzy, ILC, ball on plate system, trajectory tracking, asymptotic stability

Introduction Subramanian et al. (2017) proposed an adaptive propor-

tional–integral–derivative (PID) control scheme using model
Visual servo control, which fuses the concepts from several reference adaptive control (MRAC) technique to address the
engineering fields ranging from control theory, real-time com- plant perturbations and exogenous disturbances. The interest-
puting, image processing and kinematics, aims to enhance the ing feature of their work is that the uniform ultimate bound-
control performance by utilizing the visual information edness of tracking error is guaranteed through Schwarz’s
acquired from camera (Hutchinson et al., 1996; Mehta et al., inequality. Nevertheless, the performance of the controller
2019). Versatility, non-invasive measurement and accuracy was not tested for realistic test scenarios, such as varying light-
are the key factors that motivate visual feedback in control ing conditions and time-varying trajectory. Wang et al. (2014)
applications. Some of the practical applications of visual presented a linear control scheme to deal with the aftereffects
servo control include eye-in robot system (Barth et al., 2016), of friction using a reduced order disturbance observer.
factory automation (Mangat et al., 2021) and autonomous However, the robustness of the control scheme against exo-
vehicle (Allibert et al., 2019). The ball on plate system (BPS), genous disturbance was not evaluated in their work. To deal
which is the extension of the two-dimensional ball and beam with the friction disturbance, Wang et al. (2021) presented an
system, is a benchmark visual servo system used in several active loading control design that combined the feed-forward
universities to demonstrate and investigate the control chal-
lenges in visual feedback control. The BPS is a highly non-
1
linear, multivariable, under-actuated and open-loop unstable Department of Electrical and Electronics Engineering, Velagapudi
system. Therefore, controller design for stabilization and pre- Ramakrishna Siddhartha Engineering College, India
2
School of Electrical Engineering, Vellore Institute of Technology, India
cise trajectory tracking is considered to be a challenging prob-
lem to solve. In literature, several control techniques have Corresponding author:
been put forward to address the stabilization and reference Vinodh Kumar Elumalai, School of Electrical Engineering, Vellore Institute
tracking of BPS. Some of the notable control schemes for of Technology, Vellore 632014, India.
BPS reported in the literature are as follows. Email: [email protected]
302 Transactions of the Institute of Measurement and Control 46(2)

and feed-back proportional–derivative (PD) control with the 2. Using the Lyapunov function candidate, the global
disturbance observer. Testing the performance on a Bowden asymptotic stability of the proposed scheme is proved
cable transmission, they analysed the conditions for inner using LMIs. Moreover, the boundedness of the track-
loop stability. In another study, Armendariz et al. (2010) put ing error and control sequence is guaranteed using the
forward a hierarchical fuzzy neural network controller to singular value condition.
address the nonlinearity in BPS. Even though the efficacy of 3. Through experimental validation on a two degrees of
the control scheme through simulation study was proven, the freedom (DoF) BPS, the significant improvement in
paper failed to report the feasibility of the control scheme the robustness and tracking feature of the proposed
through the experimental validation. Comparing the perfor- TS fuzzy-ILC scheme is evaluated across the frequency
mances of sliding mode controller (SMC) and fractional-order spectrum using the cumulative power spectral density
sliding mode controller (FOSMC) for trajectory control of (CPSD) analysis.
BPS, Roy et al. (2018) showed that FOSMC could offer better
speed of response and minimal chattering compared to SMC. The remainder of the paper is organized as follows. The
Although the efficacy was tested for several input trajectories, ‘Problem formulation’ section gives the problem formulation.
the robustness against the lighting conditions and stochastic The ‘CCF-ILC augmented with TS fuzzy controller for track-
initial conditions (SICs) was not assessed. In this direction, ing’ section presents the proposed fuzzy augmented ILC
some of the other control techniques implemented for BPS scheme and explains the stability and convergence of closed-
are deep learning-based PID (Okafor et al., 2021), cascaded loop system. The ‘Application: Ball balancer’ section gives the
SMC (Das and Roy, 2017), fractional order PID (Borah control framework for the BPS and the nonlinear dynamical
et al., 2018) and neural network controller (Mohammadi and model based on Euler–Lagrangian energy-based approach.
Ryu, 2020). The ‘Experimental results and discussion’ section explains the
In spite of the aforementioned research efforts to improve three experimental test cases considered for assessing the
the tracking performance of BPS, the potential benefits of robustness of the proposed scheme. Finally, the ‘Conclusion’
augmenting learning control with feedback scheme for visual section presents the concluding remarks of the paper.
servo system has largely been unexplored (Liu and Kong,
2013; Wang et al., 2017; Xiong et al., 2017). Specifically, sev-
eral of the investigations have focussed majorly on designing Problem formulation
linear controllers for their simple design and easy parameter
tuning. However, it is important to note that the performance Consider a nonlinear system represented as a TS fuzzy model
of linear controllers is always limited because of the so-called as follows
‘waterbed’ effect, discovered by Bode. According to Bode’s
sensitivity integral, improving the tracking feature using lin- Rk : IF zi 1 (t) is Mk1 and, :::, zi n (t) is Mkn THEN
8 i
ear controller in low-frequency region inevitably improves the > i i
<x_ (t) = Fk x (t) + Gk u (t)
sensitivity to sensor noise in high-frequency region (Khalil, i i
y (t) = Ck x (t)
2002). Hence, recently there has been considerable research >
: i
attention paid to nonlinear control design to improve the con- x (0) = x0 8i, k=1, 2,    , r, i=1, 2,   
trol performance (Subramanian and Elumalai, 2018). ð1Þ
Motivated by the potentials of augmenting nonlinear fuzzy
scheme with feed-forward learning control scheme, this work where k is the fuzzy implication with r number of fuzzy rules
aims to design and investigate the TS fuzzy augmented with and the superscript i 2 N denotes the index of the iteration.
T
current cycle feedback iterative learning control (CCF-ILC) xi (t)=½xi 1 (t), xi 2 (t), :::, xi l (t)T 2 Rl , ui (t)=½ui 1 (t), ui 2 (t), :::, ui m (t)
T
scheme to improve the trajectory tracking and robustness of a 2 R and y (t)=½y 1 (t), y 2 (t), :::, y p (t) 2 R are the l–dimen-
m i i i i p

visual servo system. Specifically, to handle the nonlinear plant sional state vector, m– dimensional input vector, and p–
dynamics, we synthesize the parallel distributed compensator dimensional output vector of the system, respectively. zi n (t) is
(PDC) TS fuzzy scheme that can capture the global behaviour the premise variable at ith iteration. Fk 2 Rll , Gk 2 Rlm and
of the BPS and realize the control scheme through a velocity Ck 2 Rpl are the system state, input and output matrices,
compensation scheme. To prove the feasibility of the pro- respectively. Mkn is the fuzzy set with n number of premise
posed scheme, we not only present the theoretical formulation variables. The TS fuzzy model in equation (1) can be
using the system invariance property but also evaluate the described as
robustness of the proposed scheme experimentally using  P
hardware in loop (HIL) testing for several realistic challenges x_ i (t) = Prk = 1 hk (zi (t))½Fk xi (t) + Gk ui (t)
ð2Þ
encountered in visual servo control. In this direction, the yi (t) = rk = 1 hk (zi (t))Ck xi (t)
major contributions of this paper can be summarized as
follows. where hk (zi (t)) is the normalized membership function, which
satisfies the following criteria.
1. To solve the nonlinear stabilization and tracking con-
trol problem of under-actuated visual servo system, we mRk (zi (t))
hk (zi (t)) = Pr i
ð3Þ
present a TS fuzzy augmented ILC scheme synthesized k = 1 mRk (z (t))
based on current cycle feedback.
Jonnalagadda and Elumalai 303

Y
n
mRk (zi (t)) = Mjn (zij (t)) ð4Þ
j=1

where Mjn (zij (t)) is the membership grade of zij (t).

X
r
hk (zi (t)) ø 0 and hk (zi (t)) = 1, t 2 0, 1, 2,    ,
k=1 ð5Þ
T , k = 1, 2,    , r, 8i

where T is a positive integer. The control problem is to sepa- Figure 1. Fuzzy augmented ILC for tracking control.
rate the nonlinear system dynamics into a number of parallel
linear systems and design a PDC fuzzy controller that utilizes
the same premise variable as TS fuzzy model and find a con- can assure that the controlled output yi (t) will converge in the
trol input ui (t) to guarantee the output to track the desired neighbourhood of setpoint yd (t) even in the presence of exter-
input trajectory yd (t). nal disturbances.

Assumption 1. For a given reference trajectory yd , there exists Assumption 2. The fuzzy closed-loop system guarantees the
ud such that stability of the nonlinear system.
Assumption 3. The ILC is an add-on controller to capitalize
8 the tracking error information to enhance the tracking
< x_ d (t) = Fk xd (t) + Gk ud (t) (Gunnarsson and Norrlöf, 2001).
y (t) = Ck xd (t) ð6Þ
: d
xd (0) = x0 From Figure 1, the control input to the system is
To enhance the trajectory tracking, the fuzzy control is
ui (t) = ui FL (t) + ui ILC (t) ð7Þ
combined with the iterative learning feed-forward scheme,
which can exploit the system invariance property (Precup
where ui FL (t) is the output of the fuzzy controller and ui ILC (t)
et al., 2008). The goal of learning control scheme is to deter-
is the control signal from ILC. To defuzzify the controller out-
mine the control sequence ui (t) of system (1) using the previ-
put, the following weighted average method or centre of grav-
ous trial errors ei1 (t) and the past control inputs ui1 (t), such
ity (COG) defuzzification technique is used.
that, the output yi (t) converges to the desired reference trajec-
tory yd (t) 8 t 2 ½0, Tf  as the iteration i ! ‘. Pr i
i i k = 1 mRk (z (t)) yk d(yk )
u FL (t) = y = Pr i
ð8Þ
lim ui (t) = ud (t) ) lim yi (t) = yd (t) k = 1 mRk (z (t))
i!‘ i!‘

Substituting equation (3) into equation (8), we have the

Consequently, the tracking error ei (t) = yd (t)  yi (t) can
following fuzzy control law.
converge monotonically to zero.
X
r
lim ei (t) = 0 ui FL (t) = hj (zi (t)) Tj dxi (t) ð9Þ
i!‘
j=1

where Tj is the state feedback gain and dxi (t) = xi (t)  xi1 (t)
CCF-ILC augmented with TS fuzzy is the system state error at the ith iteration. The control signal
controller for tracking from feed-forward ILC can be represented as
ILC is a type of learning control algorithm that utilizes the
knowledge of the tracking error and control signals from pre- ui ILC (t) = ui1 (t) + Lk ei1 (t) ð10Þ
vious iterations to enhance tracking and disturbance rejection
where Lk is the learning gain. Since the fuzzy controller utilizes
features in subsequent iterations. Learning the dynamics of
the same premise variable as TS fuzzy model, the resultant
the plant over the iterations, ILC acts as an augmented con-
fuzzy-ILC control update at the (i + 1)th iteration is designed
troller on top of the feedback controller mainly to improve
as follows.
the trajectory tracking characteristics (Liu and Kong, 2013).
Figure 1 shows the schematic of CCF-ILC augmented with
the feedback fuzzy control scheme. The design of ILC can be Rk : IF zi 1 (t) is Mk1 and, :::, zi n (t) is Mkn THEN
viewed as a two-parameter synthesis problem that contains X r Xr
ui + 1 (t) = ui (t) + hk (zi (t)) hj (zi (t)) ð11Þ
the error and control effort travel in two directions, namely,
k =1 j=1
iteration k and time t (Freeman, 2017). Hence, in ILC design,
the goal is to formulate a control input sequence ui (t) that ½Lk ei (t) + Tj dxi + 1 (t)
304 Transactions of the Institute of Measurement and Control 46(2)

The condition for convergence of the fuzzy-based ILC is X

r

established in the following theorem. d_xi + 1 (t)= hk (zi (t))½Fk (xi + 1 (t)xi (t))+Gk (ui + 1 (t)  ui (t))
k =1
Theorem 1. For the nonlinear system defined in equation (1) ð20Þ
controlled by fuzzy-ILC, the update law (11) ensures that the
tracking error ei (t) has monotonic convergence. Substituting the control update ui + 1 of equation (11) into
equation (20) and from equation (13), one has
" !#
X
r X
r X
r
i+1 i i+1 i i i i i+1 i
d_x (t) = hk (z (t)) Fk dx (t) + Gk u (t) + hp (z (t)) hj (z (t))½Lp e (t) + Tj dx   u (t)
k=1 p=1 j=1

ð21Þ
lim jei (t)j = 0 as i ! ‘ ð12Þ Hence, the equation (21) becomes
i!‘

For realization, the learning gain Lk (1, 2,    ) must be X

r X
r X
r

 ðI þ Ck Gk Lk Þ\1.
chosen to satisfy the condition s d_xi + 1 (t) = hk (zi (t)) hp (zi (t)) hj (zi (t))
k=1 p=1 j=1 ð22Þ
i+1 i
Proof. To prove the convergence, we define the system state ½(Fk + Gk Tj )dx (t) + Gk Lk e (t)
error and its velocity at the current iteration as follows.
Now, substituting the equation (22) into equation (18)
gives

dxi + 1 (t) = xi + 1 (t)  xi (t)
ð13Þ X
r X
r X
r
d_xi + 1 (t) = x_ i + 1 (t)  x_ i (t) e_ i + 1 (t) = e_ i (t)  hk (zi (t)) hp (zi (t)) hj (zi (t))Ck
k =1 p=1 j=1
The tracking error and its derivative for the two consecu-
½(Fk + Gk Tj )dxi + 1 (t) + Gk Lk ei (t)
tive iterations can be expressed as
ð23Þ

ei (t) = yd (t)  yi (t)
ð14Þ From equation (1), note that dxi + 1 (0) = 08i; therefore,
e_ i (t) = y_ d (t)  y_ i (t)
equation (23) becomes
and
X
r X
r X
r
 i+1 i+1 e_ i + 1 (t) = e_ i (t)  hk (zi (t)) hp (zi (t)) hj (zi (t))
e (t) = yd (t)  y (t) ð24Þ
ð15Þ k =1 p=1 j=1
e_ i + 1 (t) = y_ d (t)  y_ i + 1 (t) i
½I + Ck Gk Lk e (t)
Combining equations (14) and (15) results in
According to the properties of the membership functions
e_ i+1 i
(t) = e_ (t)  ½_y i+1
(t)  y_ i (t) ð16Þ (Lam, 2018))

Using the output relations from equations (2) and (5), we X

r X
r X
r X
r
hk (zi (t)) = hp (zi (t)) = hj (zi (t)) = hk (zi (t))
can obtain k=1 p=1 j=1 k=1
X
r X
r
X
r
hp (zi (t)) hj (zi (t)) = 1
i+1 i i i+1 i
e_ (t) = e_ (t)  hk (z (t))Ck ½_x (t)  x_ (t) ð17Þ
p=1 j=1
k =1
ð25Þ
From equation (13), we have
Hence
X
r
e_ i + 1 (t) = e_ i (t)  hk (zi (t))Ck ½d_xi + 1 (t) ð18Þ
e_ i + 1 (t) = e_ i (t)  ½I + Ck Gk Lk ei (t) ð26Þ
k =1

Using the relations given in equations (1) and (13) Taking the norm on both sides of equation (26)

X
r j_ei + 1 (t)j ł j_ei (t)j  jI + Ck Gk Lk jjei (t)j ð27Þ
d_xi + 1 (t) = hk (zi + 1 (t))½Fk xi + 1 (t) + Gk ui + 1 (t)
k =1 It is worth noting from equation (27) that the convergence
ð19Þ
X
r
of tracking error is guaranteed when s  ½I + Ck Gk Lk \1. For
i i i
 hk (z (t))½Fk x (t) + Gk u (t)
k=1
a system controlled by ILC, when the largest singular value is
less than one, the stability is guaranteed. Consequently,
According to equation (5), the equation (9) can be simpli- e_ i + 1 (t) and e_ i (t) will converge to zero as i ! ‘. Hence, the
fied as convergence of the tracking error lim jei (t)j = 0 implies the
i!‘
Jonnalagadda and Elumalai 305

convergence of states and inputs (i.e. lim jxi (t)j = 0,

i!‘
lim jui (t)j = 0, and lim jyi (t)j = 0).
i!‘ i!‘
Theorem 2. Consider the nonlinear system described in equation
(1) controlled by the ILC update law (11) with fuzzy rules Rk .
If there exists a Lyapunov function candidate

1
V : Rn ! R, V i = (xi (t))T P(xi (t))
2
With P 2 Rnn being the symmetric positive definite
matrix, such that, PFk + FkT P\0, the global asymptotic sta-
bility of nonlinear system at the origin x0 8 i is guaranteed
when the derivative of the Lyapunov function (V_ i (xi )) is non-
increasing.
Proof. Time derivative of Lyapunov function is

1 1
V_ i (xi ) = (_xi (t))T P(xi (t)) + (xi (t))T P(_xi (t)) ð28Þ
2 2
Substituting equation (2) into equation (28) Figure 2. Two DoF ball balancer.
" #T
i 1 X r
V_ (xi ) = hk (zi (t))½Fk xi (t) + Gk ui (t) P(xi (t))
2 k=1 V_ i (xi ) ł  (xi (t))T PGk Tj dxi (t) ð36Þ
" # ð29Þ
1 i T X r
i i i Based on the LaSalle’s invariance principle, every trajec-
+ (x (t)) P hk (z (t))½Fk x (t) + Gk u (t)
2 k=1 tory that approaches the equilibrium point, that is,
xi (t) = xi1 (t), with t ! ‘ and i ! ‘ implies dxi (t) = 0.
According to the equation (25), we can obtain Therefore, equation (36) offers a sufficient condition for the
closed-loop system stability.
i 1 T
V_ (xi ) = Fk xi (t) + Gk ui (t) P(xi (t))
2 ð30Þ V_ i (xi ) ł 0 ð37Þ
1  
+ (xi (t))T P Fk xi (t) + Gk ui (t)
2 Hence, equation (37) proves that the time derivative of the
1 h i Lyapunov function (V_ i ) is non-increasing, and the closed-loop
i T
V_ (xi ) = (xi (t)) (PFk + FkT P)(xi (t)) system is globally asymptotically stable in the vicinity of the
2 ð31Þ
1 1 equilibrium at the origin for all i.
+ (xi (t))T PGk (ui (t)) + GkT (ui (t))T P(xi (t))
2 2
Equation (31) can be written as Application: Ball balancer
Figure 2 shows the two DoF ball on plate workstation that is
1h T
i
V_ i (xi ) = (xi (t)) (PFk + FkT P)(xi (t)) + (xi (t))T PGk (ui (t)) largely used in control engineering laboratories in many uni-
2 versities to teach the real-time challenges in designing a con-
ð32Þ trol scheme for visual servo system. Ball on plate workstation
is a laboratory scale model that is a highly nonlinear and mul-
According to Lyapunov stability, PFk + FkT P\0; hence
tivariable system. Moreover, it contains only two actuators to
1h i T i control the four DoF. Hence, it is also called a two DoF ball
(x (t)) (PFk + FkT P)(xi (t)) ł 0 ð33Þ balancer. The workstation comprises two servo motor
2
attached to the metal plate through gimbal arrangement. The
Substituting equation (33) into equation (32) x- and y-axes of the plate are controlled by the servo motors
and the overhead camera reads the plate image. The goal is to
V_ i (xi ) ł  (xi (t))T PGk (ui (t)) ð34Þ design a control scheme that can not only make the ball stabi-
lize on the plate but also enable the ball to track the reference
Considering the control signal of fuzzy controller trajectory even in poor lighting conditions with fewer number
expressed in equation (9), we have of actuators (i.e. servo motors).
" #
X
r
V_ i (xi ) ł  (xi (t))T PGk i i
hj (z (t))Tj dx (t) ð35Þ Ball balancer dynamics
j=1

Pr As the dynamics of x- and y-axes is identical, for brevity, we

i
Since j = 1 hj (z (t)) = 1 present the equations of motion (EoM) for the x-axis control.
306 Transactions of the Institute of Measurement and Control 46(2)

Table 1. Ball balancer system parameters.

Symbol Description Value Unit

wXd Plate size 41.75 3 41.75 cm

Lt Table length 27.5 cm
rb Ball radius 1.46 cm
mb Mass of the ball 0.003 kg
Vm Nominal voltage for servo motor 10 V
Rm Motor armature resistance 2.6 O
Lm Motor armature inductance 0.18 mH
Kt Motor torque current constant 7.68 3 103 Nm/A
Kgi Ratio of the internal gear box 14 –
vg Motor maximum speed 628.3 rad/s
rarm Distance between coupled joint and load gear 2.54 Cm
Lp Length of the support arm 14.6 Cm

_ yb  mb gyb cosb + km Vmy (t)

€ =  2mb yb b_
(mb y2b + Jb )b
Rm
0 1
ð39Þ
km2 B
B Lp b_ cos b C
 rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiC
Rm @ 2 2
1Lp sin a
A
2rarm 2
4rarm

We obtain the following nonlinear state space model of

the ball balancer by selecting the state vector as
x = ½xb , x_ b , yb , y_ b , a, a, _
_ b, b.
2 3
x_ b
6 ! 7
6 mb a_ 2 xb mb gsina 7
6 7
6 mb + 2
Ib
7
6 r
b 7
6 7
6 y_ b 7
Figure 3. Schematic diagram of ball on plate system. 6 ! 7
6 7
6 2
mb a_ yb mb gsinb 7
6 7
6 mb + 2
Ib
7
6 r
b 7
Figure 3 illustrates the schematic diagram of the ball balancer 6 7
6 a_ 0 1 7
6 7
system for x-axis control and Table 1 gives the plant para- 6
x_ = 6 7
B C 7
meters. The nonlinear dynamical equations are obtained with 6 2mb xb a_ km2
q ffiffiffiffiffiffiffiffiffiffiffiffiffi
_
Lp acosa
ffi 7
6 _ x b m gx
b b cosa Rm @ A 7
the assumption that the ball is symmetric and always main- 6 2rarm
1Lp 2 sin2 a 7
6 2
4rarm 7
tains contact with the metal plate attached to the servo motor. 6 7
6 Ip + mb x2b 7
Let xb , yb be the ball positions in x- and y-axes, respectively. 6 7
6 _b 7
aand b represent the tilt angles of the plate in the respective 6 0 1 7
6 7 ð40Þ
x- and y-axes. The gear angles of the servo motor attached to 6 7
6 2B C 7
the plate are ux and uy . The nonlinear dynamical equation of 6 2mb yb b_
_ yb mb gxb cosb m @ k
q
L p _
ffiffiffiffiffiffiffiffiffiffiffiffiffi
bcosb
ffi A 7
6 Rm 2 2 7
the ball balancer is obtained using the Euler–Lagrangian 4 2rarm
1Lp sin a
4r2
5
arm
approach. For detailed overview on modelling of two DoF Ip + mb y2b
ball balancer, interested readers can refer to (Jonnalagadda 2 3
0 0
et al., 2020) 6 0 0 7
6 7
6 7
6 0 0 7
km Vmx (t) 6 7
(mb x2b + Jb )€
a =  2mb xb a_
_ xb  mb gxb cosa + 6 0 0 7 
Rm 6 7 Vmx
0 1 +6
6 0 0
7
7 V
6 7 my
ð38Þ 6 km 7
km2 B
B _ a C
Lp a cos 6 Rm (Ip + mb x2b ) 0 7
 rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiC 6 7
Rm @ 2 2
1Lp sin a
A 6 0 0 7
2rarm 2
4 5
4rarm km
0 Rm (Ip + mb y2b )
Jonnalagadda and Elumalai 307

Table 2. Properties of membership functions. Table 3. Fuzzy rule table.

MF’s E dxb =dt u e NB NM NS Z PS PM PB

dxb =dt
s c s c s c
NB NB NB NM NM NS Z PM
NB 1.1324 28 1.5570 211 0.1698 21.2 NM NB NM NM NS Z Z PM
NM 25.3333 27.3333 20.8 NS NM NM NS NS Z PS PB
NS 22.6667 23.6667 20.4 Z NM NS NS Z PS PS PB
Z 25.55e-17 25.551e-17 0 NS NS NS Z Z PS PM PB
PS 2.6667 23.6667 20.4 PM NS Z Z PS PM PM PB
PM 5.3333 7.3333 0.8 PB Z Z Z PS PM PB PB
PB 8 11 1.2

(xc)2
mMnl (z) = exp 2s2 ð42Þ
Proposed TS fuzzy augmented ILC for BPS
Figure 4 shows the proposed ILC augmented fuzzy control (uc)2
for stabilization and reference tracking of ball balancer sys- mN1l (z) = exp 2s2 ð43Þ
tem. The cascade feedback control scheme consists of a pro-
portional controller in inner loop and TS fuzzy control in where mMnl represents the input membership function and mN1l
outer loop for load disturbance rejection and stabilization, indicates the output membership function. c and s depict the
respectively. The fuzzy controller takes two inputs, namely, mean and standard deviation of the fuzzy variable, respec-
tracking error (e) and velocity of the ball (_xb ) and provides tively. The respective I/O fuzzy sets can be written as
the required servo angle (u) to regulate the ball position (xb )
to the desired trajectory. Hence, the input vector of TS fuzzy Mnl = (z, mMnl ) : x1 2 ½8 8, x2 = ½11 11 ð44Þ
is
  N1l = (u, mN1l ) : u 2 ½45 45 ð45Þ
x1 e
x= = ð41Þ
x2 x_ b
Table 3 gives the 7 3 7 rule base of the TS fuzzy. The fir-
The constraints on e and x_ b are x1 2 ½9 9 cm and ing strength of the membership function corresponding to kth
x2 2 ½20 20 cmns. Moreover, the bounds on the servo angle rule is
to control the tilt angle of the plate are u 2 ½45 45 deg. For
the two fuzzy inputs (e and x_ b ), seven fuzzy sets, as given in mRk = mM1l (x1 ) mM2l (x2 ) ð46Þ
Table 2, are considered for fuzzification. To represent the
input-output fuzzy set, the following Gaussian membership For defuzzification, the following COG technique is
function is utilized. utilized.

Figure 4. Proposed fuzzy augmented ILC for ball on plate system.

308 Transactions of the Institute of Measurement and Control 46(2)

Figure 5. Defuzzification process using COG technique.

P
N
proposed scheme employs the following velocity compensator
mRk yk d(yk )
k =1 in the fuzzy controller.
y= ð47Þ
P
N
mRk wf s
k =1
H(s) = ð48Þ
s + wf

Figure 5 illustrates the defuzzification process using the where wf is the cut-off frequency that is experimentally chosen
COG technique that divides the aggregate area of the mem- to be 14.6 rad/seconds to counteract the sensor noise. As the
bership function into several small regions and finds the crisp stability of the BPS is guaranteed by the feedback scheme, the
value of the fuzzy output. Figure 6 illustrates the 3D output ILC capitalizes the information rich tracking error to improve
surface map that highlights the dependency of the output the command following characteristics.
(servo angle (u)) on the two inputs (error e and ball velocity
x_ b ). The surface viewer clearly highlights that when the error
increases in either direction the servo angle is proportionately Background subtraction algorithm
adjusted to retain the ball velocity within the threshold. It is Background subtraction (BS) technique is a motion detection
important to mention that, in this work, unlike the standard technique that is largely used in many visual servoing applica-
fuzzy scheme that utilizes the rate of change of error as an tions. The BS technique detects the moving objects in video
input membership function, we use the ball velocity x_ b as an stream captured by the static camera through three major
input mainly to eliminate the so-called derivative kick prob- steps, namely, (a) background initialization, (b) foreground
lem that leads to sudden rise in amplitude of controlled vari- extraction and (c) background maintenance. The basic
able (xb ). Another major reason for employing velocity approach is that the difference between the reference frame
compensation is that even a small perturbation in u due to and the current frame is computed and the pixel values that
abrupt change in xb will result in unstable system. Hence, to are higher than the threshold are classified to be foreground.
guarantee the stable system even in the presence of varying The object in the current frame is determined based on the
lighting conditions and time-varying setpoint profile, the following thresholding equation.
Jonnalagadda and Elumalai 309

Experimental results and discussion

Figure 8 shows the experimental setup, which consists of a
two DoF ball balancer workstation, a power amplifier, a
USB-based eight-channel data acquisition (DAQ) board and
a computer. The DC servo motors in the ball balancer work-
station consists of two optical encoders to measure the servo
angles (ux , uy ). The encoders have a resolution of 8192 counts/
revolution in quadrature mode. To drive the servo motors, a
power amplifier that can offer a regulated supply of 10 V at
2A is connected. The DAQ module with a resolution of 12
bits reads the servo angles at a sampling rate of 500 Hz. The
overhead camera in the ball balancer workstation captures
the 2D image of the plate at 60 frames per seconds (fps) and
uses the firewire connection for interfacing with the computer.
For HIL testing, the Quanser real-time control (QUARC)
prototype software interfaces the ball balancer workstation
with the control algorithm implemented in Simulink.

Figure 6. Surface plot.

Performance evaluation
From the 2D image acquired from the overhead camera, the
jI(x, y, t)  B(x, y, t)jT ð49Þ
BS algorithm determines the actual x and y coordinates of the
ball. Based on the actual x and y coordinates, the inner loop
where I, B and T indicate the image at time t, background or
in the control structure determines the servo angle (u) needed
reference image and the threshold value, respectively. Figure
for driving the ball to the desired trajectory. Varying the
7 depicts the sequence of action in the BS technique to deter-
servo angle results in change of plate angle (a), consequently
mine the ball coordinates. To deal with the variation in the
the ball position (xb ). Hence, from control standpoint, we can
external lighting conditions, the background model is periodi-
view this challenge as a nonlinear stabilization and tracking
cally updated as follows.
control problem. Particularly, in this work, unlike most of the
previous work in the literature that reported only the nominal
B(x, y, t + 1) = (1  g)B(x, y, t) + gI(x, y, t) ð50Þ trajectory tracking, we test the performance of the controller
for three realistic challenges of visual servo control: (a) track-
where g 2 ½0, 1 represents the learning rate that is experimen- ing under poor lighting (TPL) condition, (b) stabilization
tally chosen as 0.05. To filter out the noise in the binary under SIC and (c) following time-varying reference profile.
image, a median filter is used. Finally, the ball position in x Moreover, to evaluate the robustness, the performance of the
and y coordinates is determined based on the centriod of the proposed scheme is compared with those of three controllers,
foreground object. namely, proportional velocity (PV) controller, TS fuzzy

Figure 7. Schematic of background subtraction algorithm.

310 Transactions of the Institute of Measurement and Control 46(2)

Figure 8. Experimental test bed of ball on plate system.

controller and PV augmented ILC controller. The gains of

the PV controller determined based on pole placement tech-
nique are Kp = 3:45 and Kv = 2:11. In the ILC design, the fol-
lowing learning filter L is determined using the plant
inversion technique.

0:0248s4 + s3 + 38:25s2
L=
0:02111s4 + 0:5628s3 + 5:628s2 + 25:02s + 41:69

In the next section, we analyse the performance of the four

control schemes for the three test cases considered for
evaluation.

Test case-I: TPL condition. As visual servo system is largely

sensitive to lighting conditions, it is important to assess the
stabilization and tracking feature under poor lighting condi-
tions. Hence, a square setpoint profile of [2,2] cm with a fre-
quency of [0.07,0.06] (Hz) for the x and y coordinates is tested
under poor lighting condition. From Figure 9, which shows
the tracking response of the four control schemes, we can
note that the proposed fuzzy-ILC scheme significantly mini-
mizes the tracking error and offers smooth regulation of the
ball based on the reference profile. To highlight the improve- Figure 9. Trajectory tracking under dim lighting condition – (a) x-axis,
ment in steady state, the zoomed in view of the plot is illu- (b) y-axis.
strated. Unlike only feedback schemes (PV and fuzzy), ILC
augmented schemes offer better tracking. Specifically, com-
pared to PV-ILC scheme, the fuzzy-ILC offers significant error. Figure 10, which shows the tracking error in x- and y-
improvement in transient and steady-state regions because axes, highlights the superiority of the proposed scheme over
the fuzzy-ILC not only handles the nonlinear dynamics other control schemes considered for validation. It is also
through precise fuzzy rules but also augments appropriate important to show the precise regulation of servo angle for
control effort in every iteration to minimize the tracking controlling the tilt angle of the plate. Hence, the inner loop
Jonnalagadda and Elumalai 311

(a) (a)

(b) (b)
Figure 10. Tracking error, (a) x-axis, (b)y-axis.
Figure 12. Servo motor voltage, (Vm ) (a)x-axis, (b)y-axis.

Figure 13. Phase plot for tracking under dim lighting condition.

Figure 11. Servo angle, (a)x-axis, (b)y-axis. response than other three control schemes. Finally, to assess
both transient and steady-state tracking control performance
across the frequency spectrum, Figure 14 illustrates the
control responses of each servo angle are depicted in Figure CPSD of tracking error. We can observe that fuzzy-ILC
11. We can particularly note that ux and uy are precisely regu- offers superior performance in both low- and high-frequency
lated by the proposed scheme so as to maintain the plate spectra.
angle for balancing the ball on plate. Figure 12 that illustrates
the amount of control effort given to the servo motors high-
light that the control input (Vm ) is maintained within the Test case-II: SIC. The second experiment performed to test
nominal rating of the motors (i.e. 10 V). Moreover, to show the capability of the ILC scheme is stabilizing the ball on the
the tracking response, we present the tracking response of the plate when the ball is thrown at a controlled speed from any
all four control schemes as a phase plot in Figure 13. It is evi- direction. This kind of scenario can be viewed from control
dent that fuzzy augmented ILC scheme offers better tracking standpoint as an SIC in which the controller should be able
312 Transactions of the Institute of Measurement and Control 46(2)

Figure 14. CPSD of tracking error.

Figure 15. Tracking under stochastic initial condition (a)x-axis, (b)y-

axis.
Figure 16. Phase plot of tracking response for SIC.

to not only stabilize the ball on plate but also yield precise
positioning. Figure 15 illustrates the stabilization response of compared to other control schemes for various initial condi-
the four control schemes under SIC. It is evident from the tions and velocity of the ball.
response that the fuzzy augmented ILC scheme yields better
stabilizing and tracking response compared to other three
control schemes. Figure 16 illustrates the phase plot response Challenging trajectory (CT) tracking. Typically, in real-time
for the random initial ball position with the inset binary applications, the visual servo control should adapt to both
image depicting the ball movement. Compared to only feed- changes in setpoint magnitude and direction of input trajec-
back schemes and PV augmented ILC scheme, fuzzy-ILC tory. Therefore, as a third experiment, we evaluate the effec-
offers robust performance by stabilizing the ball on the plate tiveness of the proposed scheme to follow the time-varying
and regulating its movement close to the reference trajectory. setpoint trajectory. An input profile that as abrupt changes in
Moreover, through several experimental trials, the repeatabil- the amplitude and direction is utilized for testing. Figure 17
ity of the proposed controller to stabilize the ball for different illustrates the response of the ball movement in x- and y-axes.
speed of throw is also assessed. It is noted from the experi- To illustrate the adaptability of the control schemes to the
ments that the fuzzy-ILC scheme offers more robust response challenges in the setpoint profile, the phase plot is shown in
Jonnalagadda and Elumalai 313

(a)

(b) Figure 18. Phase plot of challenging reference trajectory tracking.

Figure 17. Challenging reference trajectory tracking, (a)x-axis, (b)y-

axis.

Figure 18. We can observe that the tracking performance of

fuzzy-ILC is better than those of the other schemes.

Performance comparison with other control

algorithms
To assess the efficacy of the proposed scheme, the tracking (a)
performance for the three test cases is quantified using the
root mean square error (RMSE) values. Table 4 gives the
RMSE of the proposed scheme with those of the similar con-
trol schemes reported in the literature on BPS, including PV,
fuzzy and PV-ILC schemes. It is evident that compared to
other three control schemes considered for benchmarking, the
proposed TS fuzzy augmented ILC scheme offers less track-
ing error and more robustness against the varying lighting
conditions and stabilization under SICs. Finally, from Figure
(b)
19, which shows the RMSE of all four control schemes for
the three tests cases in x- and y-axes, it is evident that the pro-
Figure 19. Performance comparison for the three test cases, (a)x-axis,
posed control scheme provides superior performance in all
(b)y-axis.
the three test cases considered for evaluation. Another impor-
tant point worth noting is that compared to PV and PV-ILC
schemes that considered the linear model for controller synth-
According to ‘waterbed effect’, the linear controller perfor-
esis, this work considers the nonlinearity in the BPS and
mance is always constrained due to the trade-off between the
offers better trajectory and guarantees both close setpoint
setpoint tracking and robustness. Hence, to improve the con-
tracking and disturbance rejection features.
trol performance, in this paper, we have presented a nonlinear
controller synthesized based on TS fuzzy scheme and dis-
cussed the asymptotic stability of the closed loop using
Conclusion
Lyapunov function. The motivation of this work is to present
This paper has put forward a TS fuzzy controller augmented that combining a feed-forward learning control on top of the
with the ILC to not only deal with the nonlinear plant feedback control can significantly improve the reference fol-
dynamics but also capitalize the information rich error signal lowing feature by utilizing the tracking error and control
to enhance the trajectory tracking performance of BPS. effort from previous trials. Hence, we have also presented a
314 Transactions of the Institute of Measurement and Control 46(2)

Table 4. RMSE values of the three test cases.

Reference Controller TPL SIC CT

x-axis y-axis x-axis y-axis x-axis 1 y-axis

Colmenares et al. (2014) PV 1.09 1.05 0.87 0.74 1.29 1.11

Subramanian et al. (2017) RAPID 0.98 0.95 0.72 0.69 0.82 1.02
Jonnalagadda et al. (2020) Fuzzy 0.66 0.79 0.73 0.89 0.83 1.06
Jonnalagadda and Elumalai (2020) PV-ILC 0.72 0.65 0.64 0.67 0.98 1.01
Proposed Fuzzy-ILC 0.65 0.63 0.3 0.65 0.80 0.81

monotonic error convergence of the proposed control scheme Barth R, Hemming J and van Henten EJ (2016) Design of an eye-in-
based on singular value condition. To assess the robustness of hand sensing and servo control framework for harvesting robotics
the proposed scheme, the controller performance has been in dense vegetation. Biosystems Engineering 146: 71–84.
tested for three realistic test cases. The RMSE in all three test Borah M, Roy P and Roy BK (2018) Enhanced performance in trajec-
tory tracking of a ball and plate system using fractional order con-
cases compared with the state-of-the-art methods reveal that
troller. IETE Journal of Research 64(1): 76–86.
the proposed fuzzy-ILC scheme can offer significant improve-
Colmenares SG, Armendáriz MAM, Rubio JDJ, et al. (2014) Dual
ment in tracking and robustness. In spite of the improvement PD control regulation with nonlinear compensation for a ball
in control performance, the major limitation of our approach and plate system. Mathematical Problems in Engineering 2014:
that is worth investigating is the optimal selection of ILC fil- 894209.
ter coefficients. Moreover, to handle the hard and soft con- Das A and Roy P (2017) Improved performance of cascaded
straints of the BPS, the selection of ILC filter coefficients can fractional-order SMC over cascaded SMC for position control of
be reformulated as an optimization problem and the efficacy a ball and plate system. IETE Journal of Research 63(2): 238–247.
of norm optimal ILC can be explored. Freeman CT (2017) Robust ILC design with application to stroke
rehabilitation. Automatica 81: 270–278.
Gunnarsson S and Norrlöf M (2001) On the design of ILC algorithms
Declaration of conflicting interests using optimization. Automatica 37(12): 2011–2016.
Hutchinson S, Hager GD and Corke PI (1996) A tutorial on visual
The author(s) declared no potential conflicts of interest with servo control. IEEE Transactions on Robotics and Automation
respect to the research, authorship and/or publication of this 12(5): 651–670.
article. Jonnalagadda VK and Elumalai VK (2020) Hardware-in-the-loop
testing of current cycle feedback ILC for stabilisation and track-
ing control of under-actuated visual servo system. International
Funding Journal of Systems Science 52(5): 1–19.
The author(s) disclosed receipt of the following financial sup- Jonnalagadda VK, Elumalai VK, Singh H, et al. (2020) Nonlinear
port for the research, authorship and/or publication of this control design using takagi-sugeno fuzzy applied to under-
article: This research is supported by Science and Engineering actuated visual servo system. Transactions of the Institute of Mea-
surement and Control 42(15): 2969–2983.
Research Board (SERB), India under the project scheme
Khalil HK (2002) Nonlinear Systems (3rd edn). Hoboken, NJ: Pre-
‘Teacher Associateship for Research Excellence’ (Grant No.:
ntice Hall.
TAR/2018/001187). Lam HK (2018) A review on stability analysis of continuous-time
fuzzy-model-based control systems: From membership-function-
independent to membership-function-dependent analysis. Engi-
ORCID iD
neering Applications of Artificial Intelligence 67: 390–408.
Vinodh Kumar Elumalai https://orcid.org/0000-0003- Liu X and Kong X (2013) Nonlinear fuzzy model predictive iterative
2768-3705 learning control for drum-type boiler–turbine system. Journal of
Process Control 23(8): 1023–1040.
Mangat AS, Mangler J and Rinderle-Ma S (2021) Interactive process
Supplemental material automation based on lightweight object detection in manufactur-
Supplemental material for this article is available online. ing processes. Computers in Industry 130: 103482.
Mehta SS, Ton C, Rysz M, et al. (2019) New approach to visual servo
control using terminal constraints. Journal of the Franklin Institute
References 356(10): 5001–5026.
Mohammadi A and Ryu JC (2020) Neural network-based PID com-
Allibert G, Hua MD, Krupı́nski S, et al. (2019) Pipeline following by pensation for nonlinear systems: Ball-on-plate example. Interna-
visual servoing for autonomous underwater vehicles. Control Engi- tional Journal of Dynamics and Control 8: 178–188.
neering Practice 82: 151–160. Okafor E, Udekwe D, Ibrahim Y, et al. (2021) Heuristic and deep
Armendariz MAM, Olvera CAP, Rodriguez FO, et al. (2010) Indirect reinforcement learning based PID control of trajectory tracking in
hierarchical FCMAC control for the ball and plate system. Neuro- a ball-and-plate system. Journal of Information and Telecommuni-
computing 73(13): 2454–2463. cation 5(2): 179–196.
Jonnalagadda and Elumalai 315

Precup RE, Preitl S, Tar JK, et al. (2008) Fuzzy control system perfor- Wang L, Zhu C, Yu J, et al. (2017) Fuzzy iterative learning control
mance enhancement by iterative learning control. IEEE Transac- for batch processes with interval time-varying delays. Industrial &
tions on Industrial Electronics 55(9): 3461–3475. Engineering Chemistry Research 56(14): 3993–4001.
Roy P, Das A and Roy BK (2018) Cascaded fractional order sliding Wang Y, Sun M, Wang Z, et al. (2014) A novel disturbance-observer
mode control for trajectory control of a ball and plate system. Trans- based friction compensation scheme for ball and plate system. ISA
actions of the Institute of Measurement and Control 40(3): 701–711. Transactions 53(2): 671–678.
Subramanian RG and Elumalai VK (2018) Discrete-time setpoint- Wang Z, Rho S, Yang C, et al. (2021) Active loading control design
triggered reset integrator design with guaranteed performance and for a wearable exoskeleton with a bowden cable for transmission.
stability. ISA Transactions 81: 155–162. Actuators 10(6): 108.
Subramanian RG, Elumalai VK, Karuppusamy S, et al. (2017) Uni- Xiong W, Xu L, Ho DW, et al. (2017) Synchronous and asynchro-
form ultimate bounded robust model reference adaptive PID con- nous iterative learning strategies of T–S fuzzy systems with mea-
trol scheme for visual servoing. Journal of the Franklin Institute surable and unmeasurable state information. IEEE Transactions
354(4): 1741–1758. on Fuzzy Systems 26(5): 3042–3053.