3782 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 27, NO.

8, AUGUST 2018

Single-Image Super-Resolution Based

on Rational Fractal Interpolation
Yunfeng Zhang, Qinglan Fan, Fangxun Bao, Yifang Liu, and Caiming Zhang

Abstract— This paper presents a novel single-image super- Interpolation-based methods estimate the unknown pixels in
resolution (SR) procedure, which upscales a given low-resolution the HR grid by employing their known neighbors. Traditional
(LR) input image to a high-resolution image while preserving interpolation algorithms involving bi-linear and bi-cubic [2]
the textural and structural information. First, we construct a
new type of bivariate rational fractal interpolation model and interpolation are the most widely used methods in practice.
investigate its analytical properties. This model has different However, the kernel functions used in the above methods
forms of expression with various values of the scaling factors are isotropic and cannot fully reflect the intrinsic structures
and shape parameters; thus, it can be employed to better of images. Thus, these interpolation approaches are prone to
describe image features than current interpolation schemes. producing zigzagging artifacts along edges and blurring details
Furthermore, this model combines the advantages of rational
interpolation and fractal interpolation, and its effectiveness is in textures. In order to compensate for the deficiencies of tra-
validated through theoretical analysis. Second, we develop a ditional methods, edge-directed interpolation methods [3], [6]
single-image SR algorithm based on the proposed model. The LR have recently been proposed. Although these types of methods
input image is divided into texture and non-texture regions, and can maintain the image edge structure, they often generate
then, the image is interpolated according to the characteristics of speckle noise or distortion around texture areas. The rational
the local structure. Specifically, in the texture region, the scaling
factor calculation is the critical step. We present a method to function that approximates the ideal kernel function [7] has
accurately calculate scaling factors based on local fractal analysis. been applied in image interpolation [8], [9]. Reconstructed
Extensive experiments and comparisons with the other state-of- images often have better visual results through rational func-
the-art methods show that our algorithm achieves competitive tion interpolation; however, the rational interpolation function
performance, with finer details and sharper edges. results in an unsatisfactory level in preserving textural details.
Index Terms— Image super-resolution, rational fractal inter- Learning-based SR methods depend on the assumption that
polation, image features, scaling factor, local fractal analysis. the high-frequency information missing in the LR image can
be learned from a training set of HR and LR image pairs.
I. I NTRODUCTION
Learning-based methods can be divided into two categories:

T HE purpose of single-image super-resolution (SR) is to

reconstruct a latent high-resolution (HR) image using
a single low-resolution (LR) input. SR is a classic method
the first depends on an external dictionary constructed from
a set of external training images, and the second replaces the
external training set with the LR image itself. For the former,
of image processing which has value in both academic and a HR image is generated from a single LR image with the
industrial applications. SR has a wide range of practical help of training pairs. The commonly used methods include
applications, such as video surveillance, criminal investigation, regression-based methods [10], [11], sparse representation-
remote sensing, medical image processing, and consumer based approaches [12], [13], patch-based SR methods [14],
electronics [1]. In general, existing SR methods can be broadly among others. Furthermore, anchored neighborhood regression
classified into two main categories: interpolation-based (ANR) [15] and simple functions (SF) [20] provide state-of-
methods and learning-based methods. the-art quality performance. Combining the best qualities of
Manuscript received May 9, 2017; revised November 3, 2017 and ANR and SF, an adjusted anchored neighborhood regression
January 31, 2018; accepted April 8, 2018. Date of publication April 12, 2018; (A+) [17] method was proposed for fast SR. In recent years,
date of current version April 26, 2018. This work was supported in part by the deep learning technology has attracted considerable atten-
National Natural Science Foundation of China under Grant 61373080, Grant
61672018, Grant 61402261, and Grant U1609218, and in part by the Fostering tion from numerous researchers. In image SR reconstruction,
Project of Dominant Discipline and Talent Team of Shandong Province Higher a deep learning approach (SRCNN) [16] was presented and
Education Institutions. The associate editor coordinating the review of this directly learned an end-to-end mapping between the LR and
manuscript and approving it for publication was Prof. Ce Zhu. (Corresponding
author: Fangxun Bao.) HR images. Although such methods, which rely on an external
Y. Zhang and Q. Fan are with the Department of Computer Science and dataset, typically perform well for some classes of images,
Technology, Shandong University of Finance and Economics, Jinan 250014, they understandably have a considerable drawback: they are
China (e-mail: [email protected]; [email protected]).
F. Bao is with the Department of Mathematics, Shandong University, Jinan fixed and are thus not adapted to the input image [18]. For
250100, China (e-mail: [email protected]). example, irregularities along the edges can be observed as in
Y. Liu is with the Department of Computer Science and Engineering, the results of [14], when the image content does not match
University at Buffalo, The State University of New York, Buffalo, NY
14260 USA (e-mail: [email protected]). the exemplary data set. In the latter case, the methods produce
C. Zhang is with the Department of Computer Science and Technology, HR images by employing similarity redundancy information
Shandong University, Jinan 250101, China (e-mail: [email protected]). within both the same sale and across different scales. In [18],
Color versions of one or more of the figures in this paper are available
online at http://ieeexplore.ieee.org. for each LR patch, similar self-examples were found, and
Digital Object Identifier 10.1109/TIP.2018.2826139 it was necessary to learn a linear function to directly map
1057-7149 © 2018 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted,
but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
ZHANG et al.: SINGLE-IMAGE SR BASED ON RATIONAL FRACTAL INTERPOLATION 3783

Fig. 1. Algorithm framework.

each LR patch into its HR version. However, in [18], the Note that the above fractal analytical methods characterize
LR-HR patch pairs were found by searching only for “trans- the textural feature of the image primarily by using the fractal
lated” versions of patches in the down-scaled images. A self- dimension. The fractal dimension is employed to describe the
similarity-driven SR algorithm (selfExSR) [19] was proposed roughness of the texture, but it cannot accurately characterize
to expand the internal patch search space by allowing geomet- textural details. Thus, it is necessary to construct a fractal
ric variations, which improves the visual effect well. However, function and apply related properties to accurately describe
if the LR image does not contain sufficient repetitive patterns, the image details.
then these algorithms tend to produce sharp edges rather than In this paper, we construct a rational fractal interpolation
fine details. Considering the advantages of interpolation-based model. On the one hand, compared with other polynomial
methods and learning-based methods, by applying the machine interpolation kernel functions, the rational interpolation func-
learning method in image interpolation, the study in [21] tion is a more accurate approximation function for the ideal
developed an image interpolation method in which nonlocal interpolation. It can preserve image edge structures well.
autoregressive modeling (NARM) was embedded in the sparse On the other hand, the fractal function is an efficient model
representation model. This method improves the visual effect for describing image texture, and the reconstructed images
well, but it tends to significantly degrade the quality of the have fine textural details and structural information when using
reconstructed HR image when larger magnification ratios are rational fractal function interpolation. Based on this model,
performed, particularly image details and textural features. a novel single-image SR algorithm is proposed in which the
Fractal is an efficient tool for describing the texture of an fractal analysis method is applied in image interpolation. The
image, and is widely applied in texture description, classifica- major steps of the proposed algorithm are illustrated in Fig. 1.
tion, segmentation, among other functions. In [22], a texture First, according to the image features, the image is divided into
multifractal spectrum was introduced. Based on multifractal a texture region and a non-texture region. Second, we employ
analysis in wavelet pyramids of texture images, the study the rational fractal interpolation model in the texture region,
in [23] presented a textural descriptor that implicitly com- and we use the rational interpolation model in the non-
bines information from both spatial and frequency domains. texture region. Finally, a HR image is obtained by pixel
A dynamic texture classification was proposed on the basis mapping.
of dynamic fractal analysis in [24]. A method for HR optical By applying the rational fractal interpolation model in
image segmentation was proposed based on the multifractal image SR reconstruction, the advantages of our method over
characterization of an image in [25]. In addition, a fractal other SR methods are as follows. First, it can recover more
dimension-invariant filtering method [26] was presented for pleasing details than other interpolation methods. Second,
edge detection. In [27], a depth up-sampling algorithm which unlike learning-based methods which depend on the source
jointly uses local fractal analysis and boundary consistency of training patches, our method can achieve competitive per-
analysis was proposed. However, only a limited amount of formance by only using LR image patch information.
literature on applying fractal analysis in image SR has been The major contributions of the proposed method are sum-
available until now. For example, in [28], an image SR marized as follows:
algorithm was proposed using a special type of orthogonal (1) We construct a rational fractal interpolation model based
fractal coding method which can produce pleasing details, but on previous research on rational splines [30], [33], and we
fails to recover sharper edges. In [29], a texture enhancement investigate its analytical properties, such as error analysis,
algorithm which uses local fractal analysis for improving stability, quasi-locality, and fractal dimension.
single-image SR performance was presented, and this can (2) Inspired by the constructed rational fractal interpolation
effectively enhance image details. However, this algorithm model, we propose a single-image SR method by apply-
cannot provide satisfactory results in a stochastic texture ing the fractal analysis method in the interpolation model.
region which does not follow the local fractal property. This method can accurately recover the spatial characteristic
3784 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 27, NO. 8, AUGUST 2018

information of the image, particularly image details and tex-

tural features.
(3) The extensions have two aspects: the first is con-
sideration of the relationship between scaling factors and
fractal dimension, and the second is consideration of the local
characteristics of the image. Here, a method is presented to
accurately calculate the scaling factors.
The remainder of this paper is organized as follows.
Section II will develop a new type of bivariate rational fractal
interpolation model, and it will present relevant theoretical
analyses. Section III will present the proposed single-image
SR reconstruction algorithm. The experimental results and
analysis will be presented along with an evaluation of the
effectiveness of the algorithm in Section IV. Section V will
present some conclusions.

II. P ROPOSED F RAMEWORK

A. Basic Idea
Fig. 2. Illustration of the iteration system.
The fractal texture can describe the spatial characteristics
information of the image correctly; moreover, the rational 2) Construction of Rational FIFs: Based on our previous
function is an approximation of the sinc function, which work on bivariate rational interpolation, a novel method for
corresponds to ideal filtering. Based on the general char- constructing bivariate rational FIFs is proposed in this section.
acteristics described above, we construct a rational fractal The core idea of the method is to treat continuous functions
interpolation model, and we provide the relevant theoretical qi, j (x, y) in (1) as a “fractal perturbation” of bivariate rational
analysis. The error analysis results prove that the proposed interpolation functions Pi, j (φi (x), ϕ j (y)) obtained via rational
model is good in terms of the approximation effect, and it can perturbation base functions Bi, j (x, y), namely,
control interpolation error. The stability analysis indicates that
the proposed model has fairly strong adaptability in image qi, j (x, y) = Pi, j (φi (x), ϕ j (y)) − si, j Bi, j (x, y).
interpolation. According to the property of quasi-locality,
Thus, the IFS (1) determines a continuous function (x, y),
the model is suitable for describing the local features of an
which is called a bivariate rational FIF, and
image. The fractal dimension reflects the complexity of the
texture; furthermore, it has a strong correlation with the scaling (φi (x), ϕ j (y)) = si, j (x, y) + Pi, j (φi (x), ϕ j (y))
factors. Here, we provide the method for calculating the fractal
− si, j Bi, j (x, y). (2)
dimension.
Based on the above, the proposed interpolation model Furthermore, if the shape parameters satisfy a suitable mild
has greater flexibility and better practicality than previous condition, the fractal function (x, y) defined by (2) can be
models. In the following sections, the method of constructing rewritten in the following simple matrix form:
the bivariate rational fractal interpolation function will be
provided. In essence, our constructed rational function is (φi (x), ϕ j (y)) = si, j (x, y) + AE B, (3)
embedded into an iterated function system. Then, the relevant
where A is a 4-dimensional row vector, B is a 4-dimensional
theoretical analyses of the model will be presented.
column vector, and E is a constant matrix of order 4. The
specific construction of the bivariate rational FIF is shown in
B. Rational Fractal Interpolation Function Appendix B.
Equation (2) shows that the bivariate rational FIF (x, y)
1) Iterated Function System: Fractal interpolation functions is given by an iterative scheme. In the following, we inter-
(FIFs) are types of continuous functions generated by iterated pret the iterative process of the system via a data set of a
function systems (IFSs). FIFs corresponding to the following 5 × 5 grid: {(x i , y j , fi, j ) : i, j = 1, 2, · · · , 5}. For each i
IFS form have been investigated the most extensively: and j (i, j = 1, 2, 3, 4), contractive homeomorphisms φi (x)
⎧ and ϕ j (y) map intervals [x 1 , x 5 ] and [y1 , y5 ] to subinter-
⎪
⎨φi (x) = ai x + bi , vals [x i , x i+1 ] and [y j , y j +1 ], and 5 points are obtained on
ϕ j (y) = c j y + d j , (1) [x i , x i+1 ] and [y j , y j +1 ] respectively, as shown in Fig. 2(a).
⎪
⎩
Fi, j (x, y, z) = si, j z + qi, j (x, y), Meanwhile, 5 × 5 interpolation points are provided on each
subregion [x i , x i+1 ] × [y j , y j +1 ], and 5 × 5 points on the
where |si, j | < 1 and si, j are called vertical scaling factors interpolation surface corresponding to this subregion can be
and qi, j (x, y) are continuous functions. The detailed process acquired by the iterative scheme (2); take [x 3 , x 4 ] × [y2 , y3 ]
is shown in Appendix A. as an example, as shown in Fig. 2(b), where the top and
ZHANG et al.: SINGLE-IMAGE SR BASED ON RATIONAL FRACTAL INTERPOLATION 3785

TABLE I
D ATASET TAKEN F ROM F UNCTION (4)

bottom parts of Fig. 2(b) represent interpolation surfaces and Fig. 3. Error surface f (x, y) − (x, y).
interpolation regions respectively. Thus, after the first iteration,
we obtain 17 × 17 points on region [x 1, x 5 ] × [y1 , y5 ] and the TABLE II
interpolation surface corresponding to the region. The same T HE P ERTURBATION OF THE D ATA IN TABLE 1
occurs with more iterations.
Remark 1: The construction of the rational FIF described
here allows us to embed shape parameters within the structure
of the fractal function, and the rational FIF (x, y) defined
by (2) is uniquely identified from the interpolation data and
the values of the scaling factors and shape parameters. With
different scaling factors and shape parameters, this fractal
function can be expressed in different forms, specifically, if the spline FIFs provide a good approximation for the original
scaling factors si, j = 0 for all i and j , (x, y) reduces to the function f (x, y). This result confirms that the proposed ratio-
bivariate rational spline interpolant Pi, j (x, y). This means that nal fractal model can guarantee the quality of interpolated
the presented bivariate rational fractal interpolation is a flexible images well.
and diverse form for the choice of suitable interpolant. 2) Stability: Stability is an important criterion for describ-
ing the quality of an interpolation function, and it measures
C. Theoretical Discussion the anti-interference ability of the perturbation of interpolation
data. In this part, we provide this property of the developed
1) Error Analysis: In image interpolation, the accuracy of
bivariate rational FIF.
the interpolation model directly affects the quality of the ˆ = {(x i , y j , fˆi, j , d̂ ∗ , d̂i, j ) : i ∈ I; j ∈ J } be
Let  i, j
interpolated image, which is generally assessed in terms of
another given set of interpolation points, which are gen-
errors and is the key criterion for evaluating the interpolation
erated by perturbation of the vertical coordinates f i, j and
method. The smaller the error is, the higher the accuracy is.
first-order partial derivatives di,∗ j and di, j . Define an IFS
In order to show the effectiveness with which the bivariate
rational FIF (x, y) defined by (2) approximates the original ˆ
{ F̂; (ϕi (x), ψ j (y), F̂i, j (x, y, (x, y))) : i ∈ I ; j ∈ J },
function f (x, y), we provide the upper bound of the uniform where ϕi (x) and ψ j (y) are defined as above, and
error between (x, y) and f (x, y) as follows: ˆ
F̂i, j (x, y, (x, ˆ
y)) = si, j (x, y) + P̂i, j (ϕi (x), ψ j (y))
|s|∞ ∂2
f − si, j B̂i, j (x, y),
 f − ∞ ≤ M + h 2  2 ∞ C
1 − |s|∞ ∂x
∂ Pi, j P̂i, j (ϕi (x), ψ j (y)) and B̂i, j (x, y) satisfy the corresponding
∂f
+ l( ∞ +  ∞ ), join-up conditions. (x, ˆ y) is the bivariate rational FIF gen-
∂y ∂y
erated by this IFS. We therefore have
where M is a constant associated with f i, j , di,∗ j and di, j ; C 1 + |s|∞ 2h 2l
is an error constant associated with shape parameters: |s|∞ = ˆ ∞≤
 −  δf + δd ∗ + δd ,
1 − |s|∞ 1 − |s|∞ 1 − |s|∞
max{|si, j | : i ∈ I , j ∈ J }; h = max{h i : i ∈ I }; and
l = max{l j : j ∈ J }. where δ f = max{| f i, j − fˆi, j |}, δd ∗ = max{|di,∗ j − d̂i,∗ j |}, and
Example 1: Consider the interpolation data presented δd = max{|di, j − d̂i, j |}.
in Table I, which has been approximated from the following Example 2: Consider the interpolation data shown
function by taking the values truncated to four decimal places: in Table II, which is a perturbation of the data in Table I.
4 Fig. 4 provides the surface of function (x, y) − (x, ˆ y).
f (x, y) = , Fig. 4 shows that the presented bivariate rational fractal inter-
(x + 1)2 + (y + 1)2 + 1
(x, y) ∈ [0.1, 0.5; 0.1, 0.5]. (4) polation has good capacity for stability during the perturbation
of interpolation data.
The example will show the approximation effectiveness of the 3) Quasi-Locality: In practical applications, it is often nec-
developed bivariate rational fractal interpolation. essary to adjust the local shape of the interpolation surfaces
Fig. 3 presents the graph of the error function f (x, y) − without affecting the entire shape. Consequently, interpolation
(x, y), and the values of the error range from −4 × 10−3 to functions with local properties are highly competitive. Because
4 × 10−3 . This graph clearly shows that the bivariate rational FIF has some level of self-similarity, for the fractal surface
3786 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 27, NO. 8, AUGUST 2018

Fig. 4. ˆ
Perturbation error surface (x, y) − (x, y).

TABLE III
I NTERPOLATION D ATASET FOR Q UASI -L OCALITY A NALYSIS

Fig. 5. Quasi-locality of the bivariate rational spline fractal interpolation:

(a) Rational fractal surface 4 , (b) Rational fractal surface 5 with differ-
ent si, j , (c) Rational fractal surface 6 with different αi,∗ j , βi,∗ j , γi,∗ j , and
(d) Rational fractal surface 7 with different αi, j , βi, j , γi, j .
derived from FIF, each patch in a subregion will possess
global properties inherited from the entire set of interpolation 4) Fractal Dimension: The fractal dimension of surfaces
data. This implies that the shape of the entire interpolation is a measure of the irregularity of surfaces. This describes
surface may be affected if the IFS parameters are changed the roughness of the surfaces. We provide the box-counting
in a particular subregion. In this sense, it is difficult to dimension of the rational fractal surface as follows.
constrain or adjust the local shape of the fractal surface. Denote
However, in the presented bivariate rational spline fractal ⎛ ⎞
|sτ −1 (1)| |sτ −1 (1) | · · · |sτ −1 (1) |
interpolation scheme, the “height function” Pi, j (x, y) with ⎜ |sτ −1 (2) | |sτ −1 (2) | · · · |sτ −1 (2) | ⎟
⎜ ⎟
the shape parameters possesses the local shape property such S=⎜ .. .. .. ⎟ ,
that the bivariate rational FIF has good quasi-locality capacity ⎝ . . . ⎠
on the parameters in the rational IFS. This finding will be |sτ −1 (N 2 ) | |sτ −1 (N 2 ) | ··· |sτ −1 (N 2 ) | N 2 ×N 2
interpreted using the following example. where τ (i, j ) = (i − 1) × N + j represents the enumeration
Example 3: The interpolation data is shown in Table III. of set {(i, j ) : i, j = 1, 2, · · · , N}. Let the knots be equally
Take si, j = 0.02 for all i and j ; the values of all shape spaced, namely, h i = l j = b−a N , and let the interpolation points
parameters are 0.6. Fig. 5(a) shows the corresponding bivariate of every interval be neither x-collinear nor y-collinear. Then,
rational fractal surface 4 (x, y). Now, we consider the effects the box-counting dimension D of the fractal surface defined
of the scaling factors and shape parameters on the shape of by (13) is as follows:
the fractal surface. First, we change si, j to 0.15 for i, j = 1, 2,
keeping the other scaling factors and all shape parameters as 1 + log N λ, λ > N
D= (5)
in Fig. 5(a), and Fig. 5(b) provides the corresponding rational 2, λ≤ N
fractal surface 5 (x, y). Second, take αi,∗ j = βi,∗ j = γi,∗ j = 65
for i, j = 1, 2, keeping all the scaling factors and other shape where λ = ρ(S).
parameters as in Fig. 5(a), and Fig. 5(c) presents the graph Furthermore, since
of the corresponding rational fractal surface 6 (x, y). Third, N2 N N
take αi, j = βi, j = γi, j = 46 for i, j = 1, 2, keeping all the λ = ρ(S) = τ −1 (k) = si, j ,
scaling factors and other shape parameters as in Fig. 5(a), and k=1 i=1 j =1
Fig. 5(d) is the corresponding rational fractal surface 7 (x, y). equation (5) can be rewritten as follows:
Fig. 5 shows that a significant change in the shape of the ⎧
⎪
⎪
N N
 N N
fractal surface occurs in the particular subregions in which ⎨1 + log |si, j |,
N i=1 j =1 |si, j | > N,
changes of the scaling factors or shape parameters have been D= (6)
⎪
⎪ i=1 j =1
made, and the remainder pertaining to other subregions is not ⎩2, others.
extremely sensitive. This result indicates that the developed
bivariate rational fractal interpolation has a good property of Equation (6) shows that the fractal dimension of the surfaces
quasi-locality on the IFS parameters; thus, the shape of the is closely related to the values of the scaling factors in the
fractal surface can be adjusted locally without a significant corresponding bivariate rational FIFs. For the same interpola-
influence in other regions. Furthermore, compared with the tion data and values of the shape parameters, Fig. 6 provides
scaling factors, the shape parameters have a minor effect on the fractal interpolation surfaces corresponding to the different
the shape of the fractal surface. values of the scaling factors.
ZHANG et al.: SINGLE-IMAGE SR BASED ON RATIONAL FRACTAL INTERPOLATION 3787

Fig. 6. Roughness of the surfaces for different scaling factors: (a) Smooth
rational fractal surface with si, j = 0.1, and (b) Rational fractal surface with
si, j = 0.3.

Fig. 8. Texture detection results of noisy images: (a)-(d) 3 × 3 patch, (e)-(h)

5 × 5 patch and (i)-(l) 7 × 7 patch.

Fig. 7. Texture detection.

region. As shown in Fig. 7, the main procedures of this method
are as follows. First, each LR image patch (5 × 5) is divided
III. A DAPTIVE I MAGE S UPER -R ESOLUTION into 4 sub-blocks (3 × 3). Second, using the first sub-block as
R ECONSTRUCTION A LGORITHM an example, for LR image patch f m,n , m, n = 1, 2, · · · , 5,
A. Image Interpolation 5 5
f m,n
λ = m=1 n=1 ,
1) Regional Division: In general, different regions in a 5×5
single digital image have different structural characteristics. where f m,n is the pixel value at point (m, n) and λ is the
It is difficult for image interpolation to achieve better quality average value of all pixels. Let x m,n = f m,n − λ, m, n =
by using a single model. The proposed interpolation model 1, 2, 3; the image can be divided into a texture region and a
has different forms of expression, which can be used to handle non-texture region using the following formula:
different regions. For images, regional division is a key step
in the interpolation algorithm. The interpolation quality can non − textur e r egi on, x m,n ≥ 0 or x m,n ≤ 0, ∀m, n,
be directly affected by the regional division results. textur e r egi on, other wi se.
Unlike common texture detection methods, such as the (7)
Sobel and Canny operators, our objective is to detect more
detailed textures such that the proposed SR algorithm can Specifically, if all sub-blocks have a uniform sign, i.e., if a
better play the dominant role of the rational fractal interpo- 5 × 5 LR image patch does not include an isopleth, then we
lation function. Here, we introduce the isoline method [34] regard it as a non-texture region. Otherwise, the LR image
to detect the textural and structural information of the image patch is regarded as a texture region. The binarization image
for regional division. Furthermore, we have conducted an is presented to evaluate the effectiveness of the proposed
experiment to choose an image patch of suitable size for method in Fig. 9. As indicated by the experimental results,
texture detection (the smallest size of the interpolation unit is image texture details are effectively detected. The fractal can
3 × 3). As shown in Fig. 8, considering the robustness of the describe the spatial characteristics information of the image
method, a 5×5 patch in a given LR input image is treated as a correctly; therefore, the proposed texture detection method is
unit for detecting the roughness of texture because choosing a able to exert the advantages of the rational fractal interpolation
patch that is too small will result in noise sensitivity. Similarly, function.
some texture regions will be considered as non-texture regions 2) Scaling Factor Calculation: For image processing,
if the size of the image patch is too large. the scaling factors reflect the complexity of the texture.
Pixel value is the most intuitive expression of various According to the property analysis in Section II-C, the scaling
structures in generic natural images, and it also satisfies the factors play an important role in the rational fractal interpola-
attributes of the image itself. Therefore, textural details can tion function. Thus, the quality of interpolation can be directly
be detected according to differences between values of pixels. affected by the values of scaling factors. As the accuracy
If the region detected includes an isopleth, then it is regarded of scaling factors improves, greater interpolation quality is
as a texture region. Otherwise, it is regarded as a non-texture obtained. Generally, an image patch is interpolated with the
3788 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 27, NO. 8, AUGUST 2018

Fig. 9. Results of texture detection. (a) Baboon. (b) Child. (c) Lena.

Fig. 11. Rational fractal interpolation model.

Algorithm 1 Image Interpolation Process

Fig. 10. Local scaling factors.

same scaling factors in the traditional fractal interpolation

methods, which ignores the local variations (local geometric | f 1,3 − aver |
s1,2 = Sinit ial × ,
structure) of an image. Here, we present a method of calculat- sum
ing the scaling factors based on the local features of an image. | f 3,1 − aver |
s2,1 = Sinit ial × ,
This method involves two steps. First, the initial values of the sum
scaling factors are obtained for each LR image patch. Second, | f 3,3 − aver |
s2,2 = Sinit ial × ;
the scaling factors are further obtained according to the image sum
texture information. otherwise, s1,1 = s1,2 = s2,1 = s2,2 = 0.
As shown in Fig. 10, the concrete steps of the method are 3) Interpolation Algorithm: The interpolation model has
as follows. First, as in image processing, the fractal dimension different forms of expression, which can be used to address the
reflects the complexity of the texture. Equation (6) shows texture region and non-texture region separately. In the texture
that the fractal dimension has a strong correlation with the region, the interpolation is rational fractal interpolation. In the
scaling factors. It is imperative to calculate the values of the non-texture region, we use the rational interpolation (si, j = 0).
scaling factors with the help of the fractal dimension. However, Based on the above analysis for regional division, the LR
the fractal dimension D of formula (6) is the fractal dimension input image is divided into many 5 × 5 patches. According
of the entire image, and the scaling factors si, j of the LR image to block characteristics, a 5 × 5 vector control grid is used
patch cannot be obtained by using it directly. Based on the to construct the interpolation surface, and we then use it and
property that the proposed bivariate rational FIF has a good rational fractal interpolation to compute every point’s intensity
capacity of quasi-locality on the parameters in the rational in the magnified image. As shown in Fig. 11, for the given
IFS, we present an efficient method of calculating the scaling 5 × 5 image patch, a 17 × 17 patch can be obtained by using
factors with the help of the local fractal dimension [35]. the proposed model (see Section II-B). The red dots represent
Suppose that each LR image patch uses the same scaling the LR image pixels, and the missing HR samples are the
factors, where the initial values of the scaling factors Sinit ial green dots. The main steps of the interpolation algorithm are
are obtained by calculating the following formula: summarized in Algorithm 1.
The LR image is divided into 5×5 patches, and the proposed
Sinit ial = N D−3 . (8) interpolation model is used to estimate the missing pixels in
the LR image patch. In order to guarantee a smooth connection
Then, it is unsuitable to use the same scaling factor for LR
between patches, there is an overlapping pixel point between
image patches with complex geometric structures. Take the
the patches in the processing. The image interpolation process
first sub-block ( f m,n , m, n = 1, 2, 3) as an example: if it is
is shown in Fig. 12. The interpolation is extended to the entire
the texture region,
image by traversing each patch in raster-scan order in the LR
f 1,1 + f 1,2 · · · + f 3,3 image, namely from left to right and from top to bottom.
aver = ,
9
sum = | f 1,1 − aver | + | f 1,3 − aver | + | f 3,1 − aver | B. Mapping
+ | f 3,3 − aver |, The proposed rational fractal interpolation function is an
| f 1,1 − aver | IFS. A large number of pixels are obtained with an increasing
s1,1 = Sinit ial × ,
sum number of iterations, and more detailed information can be
ZHANG et al.: SINGLE-IMAGE SR BASED ON RATIONAL FRACTAL INTERPOLATION 3789

Fig. 14. The process of pixel mapping (×3).

Fig. 12. Illustration of the image interpolation process.

Fig. 13. The process of pixel mapping (×2).

captured. Thus, the image can be amplified at any integral

multiple via pixel mapping. In fact, pixel mapping is a
sampling process.
Specifically, after the first iteration, the image can be
magnified with three different scale factors (×2, ×3, and ×4). Fig. 15. Performance variation of the proposed model with different
As shown in Section II-B, for the given 5 × 5 patch, a 17 × 17 parameters α, β, γ on the training dataset.
patch can be obtained by using the proposed model. Thus,
the image with an upscaling factor of 4 can be obtained
gi+1, j +1 is obtained from the following equations:
directly by using the proposed model. The image also can be
magnified with two different scale factors (2 and 3) using the var H or = var ( f i+2, j +1 , f i+2, j +2 , f i+2, j +3 ),
pixel mapping process which will be described in the following var V er = var ( f i+1, j +2 , f i+2, j +2 , f i+3, j +2 ),
sections. As shown in Fig. 13 and Fig. 14, taking the first sub-
block ( f i, j ) as an example, the HR image sub-block (gx,y and var 45 = var ( f i+1, j +3 , fi+2, j +2 , f i+3, j +1 ),
h m,n ) can be obtained using the following: f i, j , g x,y , h m,n , var 135 = var ( f i+1, j +1 , fi+2, j +2 , f i+3, j +3 ),
the pixel values at points (i, j ), (x, y), (m, m), respectively. var Mi n = min(var H or, var V er, var 45, var 135),
As shown in Fig. 13,
where var () is a variance function and mi n() is a minimal
j +4 i+4 function. Variance reflects the difference between pixel values.
f i,t f s, j
g x,y+1 = × f i,t , gx+1,y = × f s, j , Texture direction can be obtained approximatively according
t= j
+4
j
s=i

i+4
to the minimum value of variance. Thus, gi+1, j +1 is closest
f i,t f s, j
t= j s=i to these pixels whose texture direction is consistent with that
3790 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 27, NO. 8, AUGUST 2018

TABLE IV
O BJECTIVE Q UALITY A SSESSMENT OF D IFFERENT M ETHODS (S ET 5)

of gi+1, j +1 . For example, if var Mi n = var H or , gi+1, j +1 = h m+1,n+1 is determined by 9 pixels ( f s,t , s = i, . . . , i +
f i+2, j +1 + f i+2, j +2 + f i+2, j +3
, and so on. 2, t = j, . . . , j + 2), h m+1,n+2 is determined by 9 pixels
3
As shown in Fig. 14, ( f s,t , s = i, . . . , i + 2, t = j + 2, . . . , j + 4), h m+2,n+1
j +3 is determined by 9 pixels ( f s,t , s = i + 2, . . . , i + 4, t =
f i,t j, . . . , j + 2), h m+2,n+2 is determined by 9 pixels ( f s,t , s =
h m,n+1 = × fi,t ,
+3
j
t= j i + 2, . . . , i + 4, t = j + 2, . . . , j + 4), and the method of
f i,t
t= j computation is the same as mentioned above.
j +4 Because FIF has some level of self-similarity, fine image
f i,t
h m,n+2 = × f i,t , structures can be recovered using this feature. Moreover, it is
+4
j
t = j +1 f i,t highly advantageous to improve the interpolation accuracy of
t = j +1 images using these pixels.
i+3
f s, j
h m+1,n = × f s, j ,
s=i

i+3
IV. E XPERIMENTAL AND D ISCUSSION
f s, j
s=i
i+4 We conduct a variety of experiments to evaluate the effec-
f s, j
h m+2,n = × fs, j , tiveness of the proposed SR algorithm. Thirteen classic and
s=i+1

i+4
recent state-of-the-art methods, including interpolation, sta-
f s, j
s=i+1 tistical and learning methods, are used to compare the SR
ZHANG et al.: SINGLE-IMAGE SR BASED ON RATIONAL FRACTAL INTERPOLATION 3791

TABLE V
O BJECTIVE Q UALITY A SSESSMENT OF D IFFERENT M ETHODS (S ET 14)

Fig. 16. Comparison of SR results (×2) on Girl image. (a) bicubic. (b) BMF. (c) NEDI. (d) ICBI. (e) DCCI. (f) Lanczos. (g) ScSR. (h) NARM. (i) SRCNN.
(j) A+. (k) selfExSR. (l) ANR. (m) Tai’s. (n) our method. (o) original image.
3792 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 27, NO. 8, AUGUST 2018

Fig. 17. Comparison of SR results (×2) on Wall image. (a) bicubic. (b) BMF. (c) NEDI. (d) ICBI. (e) DCCI. (f) Lanczos. (g) ScSR. (h) NARM. (i) SRCNN.
(j) A+. (k) selfExSR. (l) ANR. (m) Tai’s. (n) our method. (o) original image.

Fig. 18. Comparison of SR results (×2) on Baboon image. (a) bicubic. (b) BMF. (c) NEDI. (d) ICBI. (e) DCCI. (f) Lanczos. (g) ScSR. (h) NARM.
(i) SRCNN. (j) A+. (k) selfExSR. (l) ANR. (m) Tai’s. (n) our method. (o) original image.

performance. Interpolation-based SR methods include bicu- down-sampling the HR images directly along both the horizon-
bic interpolation, BMF (applying median filtering for image tal and vertical directions by a factor of 2, 3, or 4. For instance,
preprocessing and then performing bicubic interpolation), down-sampling with a factor of 2 means reducing a N × N
NEDI [3], ICBI [5], DCCI [6], and Lanczos interpolation. image to a N2 × N2 image by throwing away every other row and
Tai et al. [39] method is a statistical method. ScSR [13], column. On the other hand, we also test the performance of all
NARM [21], SRCNN [16], A+ [17], SelfExSR [19], and SR methods in more realistic conditions in which LR images
ANR [15] are learning techniques. Moreover, ScSR, SRCNN, are obtained by unknown downscaling operators. The human
A+, and ANR rely on an external training set. Set5 [36], visual system is more sensitive to the luminance channel than
Set14 [37], and DIV2K [38] are employed to evaluate the the chrominance channels. Thus, we transform the color RGB
performance of the single-image SR methods in this paper. images into YCbCr color space and only conduct all compared
In order to assess the SR performance obtained from different methods in the Y channel. The remaining channels (Cb and Cr)
methods, three objective assessment indices of peak signal-to- are simply magnified using the bicubic interpolation algorithm.
noise ratio (PSNR), structural similarity (SSIM), and feature All experiments are performed using software provided by the
similarity (FSIM) are reported. authors of these methods.

A. Experimental Configuration B. Parameter Settings

In order to demonstrate the improvements of the proposed The proposed rational fractal interpolation model contains
algorithm, we perform tests on the image datasets men- shape parameters that can be employed to slightly adjust the
tioned above. In our experiments, LR images are obtained by shape of the surface. The error analysis in Section II-C shows
ZHANG et al.: SINGLE-IMAGE SR BASED ON RATIONAL FRACTAL INTERPOLATION 3793

Fig. 19. Comparison of SR results (×3) on Baby image. (a) bicubic. (b) BMF. (c) Lanczos. (d) ScSR. (e) NARM. (f) SRCNN. (g) A+. (h) selfExSR.
(i) ANR. (j) Tai’s. (k) our method. (l) original image.

Fig. 20. Comparison of SR results (×3) on Raccoon image. (a) bicubic. (b) BMF. (c) Lanczos. (d) ScSR. (e) NARM. (f) SRCNN. (g) A+. (h) selfExSR.
(i) ANR. (j) Tai’s. (k) our method. (l) original image.

that the shape parameters have a minor effect in the rational Tables IV and V present the objective qualities generated
fractal interpolation function compared with the scaling fac- by all methods for the test images. In Table IV, Set5 is used
tors. For an interpolated image based on the model, the influ- to evaluate the performance of upscaling factors of 2, 3, and
ence of the shape parameters is minor for local image pixels. 4, and Set14 is used to evaluate the upscaling factor of 3
Nevertheless, we obtain the suitable range of shape parameters in Table V. Each image corresponds to three lines in the tables,
using a number of training images. Fig. 15 plots the changing PSNR, SSIM, and FSIM from top to bottom, respectively. “-”
curved lines, which vary with different shape parameters on indicates that related methods cannot achieve the function that
the training dataset. The difference is small between α and β; the image is magnified with factors of 3 and 4. It is clear
for simplicity, let α = β. Based on this figure, the shape from these results that the proposed algorithm achieves the
parameters are determined from the numbers of experiments, best results among the compared methods in terms of the
and α, β, γ are selected in [0.4, 1.2], [0.4, 1.2], [0.5, 2.5] at three quantitative assessments. Specifically, when the image is
random, respectively. Furthermore, the experimental results magnified with higher scale factors, the objective data of our
demonstrate that the range of shape parameters that we have method are higher than is achieved using other methods. For
selected is reasonable. example, our method yields the best quantitative results for test
images with an upscaling factor of 4 in Table IV, 3.3-5.7 dB
C. Quality Assessment PSNR better than other methods on average. This result occurs
In order to examine the performance of the proposed because the proposed interpolation model is IFS, and more
algorithm, we ran three versions of the algorithm for three image information can be obtained with less data, compared
different scale factors (2, 3, and 4). with other methods. Moreover, the proposed method has clear
3794 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 27, NO. 8, AUGUST 2018

Fig. 21. Comparison of SR results (×4) on Man image. (a) bicubic. (b) BMF. (c) Lanczos. (d) ScSR. (e) SRCNN. (f) A+. (g) selfExSR. (j) ANR. (i) Tai’s.
(j) our method.

Fig. 22. Comparison of SR results (×4) on Zebra image. (a) bicubic. (b) BMF. (c) Lanczos. (d) ScSR. (e) SRCNN. (f) A+. (g) selfExSR. (j) ANR.
(i) Tai’s. (j) our method.

advantages compared with other methods in the SSIM and in the proposed algorithm’s image compared to the other
FSIM, which are used for measuring the structure similarity methods, where jaggy and blurry artifacts are observed along
between two images. This result demonstrates that the fractal the edges in the magnified local regions. NARM, NEDI, and
model is quite efficient for describing image textures. our proposed method can efficiently preserve the sharpness of
For visual comparisons, Figs. 16-22 present the SR results edges.
of these methods. Figs. 16-18 show images with a resolution As shown in Figs. 19-20, overall, with severe blurring
enhancement factor of 2, Figs. 19-20 show images with a artifacts, the performance of BMF is always inferior to that
resolution enhancement factor of 3, and Figs. 21-22 show the of the other methods. Fig. 19(c) and Fig. 20(c) show the
SR results of different approaches using an upscaling factor results of the bicubic method, which fails to recover the
of 4, where LR images are obtained by unknown downscaling structural information of images. As evidenced, some fine
operators. Overall, compared with those of the other state-of- image structures are not recovered by Lanczos. ScSR suffers
the-art methods, the results of the proposed scheme are more from aliasing artifacts in the texture region. The images
appealing. The primary advantage of our approach is its ability obtained by NARM produce overly smooth edges, resulting
to obtain vivid texture regions. in a loss of the fine textures. In the SRCNN, A+, selfExSR,
As shown in Figs. 16-18, the proposed method can recover and ANR magnified images, artifacts appear, and the texture
more pleasing details than the other compared methods. The is twisted heavily. Tai’s method suffers from fuzzy artifacts.
images generated by NEDI, ICBI, and DCCI are fuzzy and Our method succeeds in avoiding these undesired effects.
distorted. Bicubic, BMF, and Tai’s methods also suffer from Additionally, the images generated by the proposed method
blurred artifacts. The results achieved by Lanczos and ScSR show an outstanding improvement on the compared methods.
show unnatural details. The texture details are not efficiently Figs. 21-22 show the SR experiments on two LR bench-
preserved in SRCNN, A+, selfExSR, and ANR. Furthermore, marks with unknown sampling kernels. A comparison of
in Fig. 16, the improvement in edge preservation is significant different SR approaches for restoring the texture details is
ZHANG et al.: SINGLE-IMAGE SR BASED ON RATIONAL FRACTAL INTERPOLATION 3795

shown in Fig. 21. Bicubic, BMF, Lanczos, ScSR, and Tai’s where 0 ≤ λ < 1. Let ϕ j (y) be contractive homeomorphisms:
methods are prone to blurring the image details. Displeasing J → Jj
artifacts along edges are produced by SRCNN and selfExSR.
ϕ j (y1 ) = y j , ϕ j (y M ) = y j +1 ,
selfExSR also suffers from blocking artifacts. ANR, A+,
and our method achieve better results among the considered |ϕ j (d1 ) − ϕ j (d2 )| ≤ μ|d1 − d2 |, ∀d1 , d2 ∈ J,
methods. As shown in Fig. 22, the proposed algorithm exhibits where 0 ≤ μ < 1. Furthermore, let F = I × J × R, and for
a visually appealing appearance compared to the comparison i ∈ I , j ∈ J , the continuous mappings Fi, j : F → R fulfill
method.
Fi, j (x 1 , y1 , z 1,1 ) = z i, j , Fi, j (x 1 , y M , z 1,M ) = z i, j +1 ,
To summarize, our method outperforms the other algo-
rithms in terms of the objective quality of the SR images. Fi, j (x N , y1 , z N,1 ) = z i+1, j , Fi, j (x N , y M , z N,M ) = z i+1, j +1 .
Furthermore, from the visual results presented, the proposed For i ∈ I , j ∈ J , define functions wi, j : F → F,
algorithm can perform better in image detail areas than the
wi, j (x, y, z) = (φi (x), ϕ j (y), Fi, j (x, y, z)), (9)
other methods, and it can retain the edge region well. There-
fore, the constructed interpolation model is quite efficient for then, IFS (9) generates a unique attractor which is a graph of
describing the space characteristics information of images. a continuous function  : I × J → R satisfying (x i , y j ) =
z i, j , function  is called a FIF, and the following relation
holds:
V. C ONCLUSION
(φi (x), ϕ j (y)) = Fi, j (x, y, (x, y)), i ∈ I , j ∈ J ,
In this paper, we present a single-image SR algorithm based
or
on the rational fractal interpolation model, which is more
suitable for describing the structures of an image. First, for (x, y) = Fi, j (φi−1 (x), ϕ −1 −1
j (y),  ◦ (φi (x).
each LR image patch, the isoline method is employed to detect FIFs corresponding to the following IFS form were investi-
texture, such that more detailed textures can be obtained, gated the most extensively:
and the LR image is divided into texture regions and non- ⎧
texture regions. Second, in the interpolation model, the scaling ⎪
⎨φi (x) = ai x + bi ,
factors play an important role, whereas the influence of the ϕ j (y) = c j y + d j , (10)
⎪
⎩
shape parameters is minor. Based on the relationship between Fi, j (x, y, z) = si, j z + qi, j (x, y),
scaling factors and the fractal dimension, the scaling factors
where
are accurately calculated by using the image local structure
x i+1 − x i x N x i − x 1 x i+1
feature. Nevertheless, a suitable range of shape parameters is ai = , bi =
obtained using a number of training images. Then, rational x N − x1 x N − x1
y j +1 − y j y M y j − y1 y j +1
fractal interpolation and rational interpolation are used in cj = , dj = ,
the texture region and the non-texture region, respectively. y M − y1 y M − y1
Specifically, each LR image patch is first interpolated, and with −1 < si, j < 1. The parameters si, j are called the vertical
the interpolation is extended to the entire image by traversing scaling factors.
each patch. Finally, an HR image is obtained by pixel mapping.
Because the proposed rational fractal interpolation function is B. Construction of Bivariate Rational FIFs
an IFS, the image can be amplified at any integral multiple Let  = [a, b; c, d] be the plane region,  =
by selecting a suitable mapping. The experimental results {(x i , y j , fi, j , di,∗ j , di, j ) : i ∈ I; j ∈ J } a given set of
demonstrate that the proposed algorithm achieves competitive data points, where a = x 1 < x 2 < · · · < x N = b and
performance and generates high-quality SR images with sharp c = y1 < y2 < · · · < y M = d are the knot spacings, f i, j
edges and rich texture. represents f i, j (x, y) at the point (x i , y j ), and di,∗ j and di, j are
the chosen first-order partial derivatives ∂ f ∂(x,y) x and ∂ f (x,y)
∂y
at the knots (x i , y j ), respectively. Denote h i = x i+1 − x i ,
A PPENDIX
l j = y j +1 − y j , H N = x N − x 1 , L M = y M − y1 , and for any
A. Iterated Function System (IFS) point (x, y) ∈ , θ = xx−x 1
N −x 1
, η = yy−y
M −y1
1
.
Denote
Let {(x i , y j , z i, j ), i = 1, 2, . . . , N; j = 1, 2, . . . , M} be
a given set of data points, closed interval I = [a, b] ⊆ R (1 − θ )2 (αi,∗ j + θ γi,∗ j )
ω0,0 (θ, •∗i, j ) = ,
contains {x 1 , x 2 , · · · , x N }, and closed interval J = [c, d] ⊆ (1 − θ )2 αi,∗ j + θ (1 − θ )γi,∗ j + θ 2 βi,∗ j
R contains {y1 , y2 , · · · , y M }. Set Ii = [x i , x i+1 ] for i ∈ θ 2 (βi,∗ j + (1 − θ )γi,∗ j )
I = {1, 2, · · · , N − 1} and J j = [y j , y j +1 ] for j ∈ ω0,1 (θ, •∗i, j ) = ,
J = {1, 2, · · · , M − 1}. Denote I = {1, 2, · · · , N}, J = (1 − θ )2 αi,∗ j + θ (1 − θ )γi,∗ j + θ 2 βi,∗ j
{1, 2, · · · , M}. Let φi (x) be contractive homeomorphisms: I θ (1 − θ )2 αi,∗ j
→ Ii , ω1,0 (θ, •∗i, j ) = ,
(1 − θ )2 αi,∗ j + θ (1 − θ )γi,∗ j + θ 2 βi,∗ j
φi (x 1 ) = x i , φ(x N ) = x i+1 , θ 2 (1 − θ )βi,∗ j
ω1,1 (θ, •∗i, j ) = .
|φi (c1 ) − φi (c2 )| ≤ λ|c1 − c2 |, ∀c1 , c2 ∈ I, (1 − θ )2 αi,∗ j + θ (1 − θ )γi,∗ j + θ 2 βi,∗ j
3796 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 27, NO. 8, AUGUST 2018

We construct bivariate rational spline interpolants as fol- then, the IFS { × R; (φi (x), ϕ j (y), Fi, j (x, y, z))} defined by
lows: first, the x-direction interpolating curve Pi,∗ j (φi (x)) in (12) admits a unique attractor G, and G is the graph of a
[x 1 , x N ] is defined by: continuous function (x, y), and
1 (φi (x), ϕ j (y)) = si, j (x, y) + Pi, j (φi (x), ϕ j (y))
Pi,∗ j (φi (x)) = [ω0,r (θ, •∗i, j ) f i+r, j + h i ω1,r (θ, •∗i, j )di+r,
∗
j ], − si, j Bi, j (x, y). (13)
r=0
Denote:
where αi,∗ j > 0, βi,∗ j > 0, and γi,∗ j > 0. The interpolant
Pi,∗ j (φi (x)) satisfies: A = (ω0,0 (θ, •∗i, j ), ω0,1 (θ, •∗i, j ), ω1,0 (θ, •∗i, j ), ω1,1 (θ, •∗i, j )),
B = (ω0,0 (η, •i, j ), ω0,1 (η, •i, j ), ω1,0 (η, •i, j ), ω1,1 (η, •i, j ))T ,
Pi,∗ j (xr ) = fr, j , Pi,∗ j (xr ) = dr,∗ j , r = i, i + 1. ⎛ ⎞
E 0,0 E 0,1 D0,0 D0,1
Then, for i ∈ I , j ∈ J and αi, j > 0, βi, j > 0, γi, j > 0, ⎜ E 1,0 E 1,1 D1,0 D1,1 ⎟
E =⎜ ∗
⎝ D0,0 ∗
⎟,
using the x-direction interpolation function Pi,∗ j (φi (x)) defines D0,1 0 0 ⎠
∗ ∗
the bivariate rational interpolation function on  as follows: D1,0 D1,1 0 0
1 where for r, s = 0, 1,
Pi, j (φi (x), ϕ j (y)) = [ω0,s (η, •i, j )Pi,∗ j +s (φi (x))
s=0
Er,s = fi+r, j +s − si, j fr(N−1)+1,s(M−1)+1 ,
∗ ∗ ∗
+ l j ω1,s (η, •i, j )Di, j +s (φi (x))], (11) Dr,s = h i di+r, j +s − si, j H N dr(N−1)+1,s(M−1)+1 ,
Dr,s = l j di+r, j +s − si, j L M dr(N−1)+1,s(M−1)+1 .
where
When the shape parameters satisfy •∗i, j = •∗i, j +1 , the fractal
Di, j (φi (x)) = ω0,0 (θ, •∗i, j )di, j + ω0,1 (θ, •∗i, j )di+1, j . surface (x, y) defined by (13) can be rewritten as the
following simple matrix form:
The interpolant Pi, j (x, y) satisfies:
(φi (x), ϕ j (y)) = si, j (x, y) + AE B. (14)
∂ Pi, j (xr , ys ) ∗
Pi, j (xr , ys ) = fr,s , = dr,s ,
∂x
∂ Pi, j (xr , ys ))
= dr,s , r = i, i + 1; s = j, j + 1. R EFERENCES
∂y
[1] S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image
Motivated by the construction method of bivariate rational reconstruction: A technical overview,” IEEE Signal Process. Mag.,
spline interpolation, rational perturbation base functions are vol. 20, no. 3, pp. 21–36, May 2003.
[2] H. S. Hou and H. Andrews, “Cubic splines for image interpolation and
defined as follows: digital filtering,” IEEE Trans. Acoust., Speech, Signal Process., vol. 26,
1 no. 6, pp. 508–517, Dec. 1978.
[3] X. Li and M. T. Orchard, “New edge-directed interpolation,” IEEE
Bi, j (x, y) = [ω0,s (η, •i, j )Bs(M−1)+1,i, j +s (x) Trans. Image Process., vol. 10, no. 10, pp. 1521–1527, Oct. 2001.
s=0 [4] L. Zhang and X. Wu, “An edge-guided image interpolation algorithm
+ L M ω1,s (η, •i, j )K s(M−1)+1,i, j +s (x)], via directional filtering and data fusion,” IEEE Trans. Image Process.,
vol. 15, no. 8, pp. 2226–2238, Aug. 2006.
[5] A. Giachetti and N. Asuni, “Real-time artifact-free image upscaling,”
where IEEE Trans. Image Process., vol. 20, no. 10, pp. 2760–2768, Oct. 2011.
1 [6] D. Zhou, X. Shen, and W. Dong, “Image zooming using directional
cubic convolution interpolation,” IET Image Process., vol. 6, no. 6,
Bk,i, j (x) = [ω0,r (θ, •∗i, j ) fr(N−1)+1,k pp. 627–634, Aug. 2012.
r=0 [7] P. Thévenaz, T. Blu, and M. Unser, “Image interpolation and resam-
+ H N ω1,r (θ, •∗i, j )dr(N−1)+1,k
∗
], pling,” in Proc. Handbook Med. Imag., 2000, pp. 393–420.
[8] S. Carrato and L. Tenze, “A high quality 2× image interpolator,” IEEE
∗ ∗
K k,i, j (x) = ω0,0 (θ, •i, j )d1,k + ω0,1 (θ, •i, j )d N,k . Signal Process. Lett., vol. 7, no. 6, pp. 132–134, Jun. 2000.
[9] Y. Liu, Y. Zhang, Q. Guo, and C. Zhang, “Image interpolation based
Then, function Bi, j (x, y) satisfies on weighted and blended rational function,” in Proc. ACCV Workshops
Comput. Vis., Apr. 2014, pp. 78–88.
[10] K. I. Kim and Y. Kwon, “Example-based learning for single-image
∂ Bi, j (xr , ys ) ∗
Bi, j (xr , ys ) = fr,s , = dr,s , super-resolution,” in Pattern Recognition (Lecture Notes in Computer
∂x Science), vol. 5096. Berlin, Germany: Springer, Jun. 2008, pp. 456–465.
∂ Bi, j (xr , ys ) [11] Y. Tang, Y. Yuan, P. Yan, and X. Li, “Greedy regression in sparse
= dr,s , r = 1, N; s = 1, M. coding space for single-image super-resolution,” J. Vis. Commun. Image
∂y Represent., vol. 24, no. 2, pp. 148–159, Feb. 2013.
[12] J. Yang, J. Wright, T. Huang, and Y. Ma, “Image super-resolution
Now, consider the following IFS: as sparse representation of raw image patches,” in Proc. IEEE Conf.
⎧ Comput. Vis. Pattern Recognit., Jun. 2008, pp. 1–8.
⎪
⎪ φi (x) = ai x + bi , [13] J. Yang, J. Wright, T. S. Huang, and Y. Ma, “Image super-resolution
⎪
⎨ϕ (y) = c y + d , via sparse representation,” IEEE Trans. Image Process., vol. 19, no. 11,
j j j pp. 2861–2873, Nov. 2010.
(12)
⎪
⎪ Fi, j (x, y, z) = si, j z + Pi, j (φi (x), ϕ j (y)) [14] Y. HaCohen, R. Fattal, and D. Lischinski, “Image upsampling via
⎪
⎩ texture hallucination,” in Proc. IEEE Int. Conf. Comput. Photograph.,
−si, j Bi, j (x, y), Mar. 2010, pp. 1–8.
ZHANG et al.: SINGLE-IMAGE SR BASED ON RATIONAL FRACTAL INTERPOLATION 3797

[15] R. Timofte, V. De Smet, and L. Van Gool, “Anchored neighborhood [39] Y.-W. Tai, S. Liu, M. S. Brown, and S. Lin, “Super resolution using edge
regression for fast example-based super-resolution,” in Proc. IEEE Int. prior and single image detail synthesis,” in Proc. IEEE Conf. Comput.
Conf. Comput. Vis., Dec. 2013, pp. 1920–1927. Vis. Pattern Recognit. (CVPR), Jun. 2010, pp. 2400–2407.
[16] C. Dong, C. C. Loy, K. He, and X. Tang, “Learning a deep convolutional
network for image super-resolution,” in Proc. Eur. Conf. Comput. Vis.
(ECCV), 2014, pp. 184–199.
[17] R. Timofte, V. De Smet, and L. Van Gool, “A+: Adjusted anchored Yunfeng Zhang received the B.E. degree in com-
neighborhood regression for fast super-resolution,” in Proc. Asian Conf. putational mathematics and application software
Comput. Vis. (ACCV), Nov. 2015, pp. 111–126. from the Shandong University of Technology, Jinan,
[18] M. Bevilacqua, A. Roumy, C. Guillemot, and M.-L. A. Morel, China, in 2000, and the M.S. degree in applied
“Single-image super-resolution via linear mapping of interpolated self- mathematics and the Ph.D. degree in computa-
examples,” IEEE Trans. Image Process., vol. 23, no. 12, pp. 5334–5347, tional geometry from Shandong University, Jinan,
Dec. 2014. in 2003 and 2007, respectievely. He is currently a
[19] J.-B. Huang, A. Singh, and N. Ahuja, “Single image super-resolution Professor with the Shandong Provincial Key Lab-
from transformed self-exemplars,” in Proc. IEEE Conf. Comput. Vis. oratory of Digital Media Technology, Shandong
Pattern Recognit. (CVPR), Jun. 2015, pp. 5197–5206. University of Finance and Economics. His cur-
[20] C.-Y. Yang and M.-H. Yang, “Fast direct super-resolution by simple rent research interests include computer-aided
functions,” in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), Dec. 2013, geometric design, digital image processing, computational geometry, and
pp. 561–568. function approximation.
[21] W. Dong, L. Zhang, R. Lukac, and G. Shi, “Sparse representation based
image interpolation with nonlocal autoregressive modeling,” IEEE Trans.
Image Process., vol. 22, no. 4, pp. 1382–1394, Apr. 2013.
[22] Y. Xu, H. Ji, and C. Fermüller, “Viewpoint invariant texture description
Qinglan Fan received the B.E. degree from the
using fractal analysis,” Int. J. Comput. Vis., vol. 83, no. 1, pp. 85–100,
School of Information and Electrical Engineering,
2009.
Ludong University, Yantai, China, in 2015. She
[23] Y. Xu, X. Yang, H. Ling, and H. Ji, “A new texture descriptor is currently pursuing the M.S. degree with the
using multifractal analysis in multi-orientation wavelet pyramid,” in Shandong Provincial Key Laboratory of Digital
Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2010, Media Technology, Shandong University of Finance
pp. 161–168. and Economics. Her research interests include
[24] Y. Xu, Y. Quan, H. Ling, and H. Ji, “Dynamic texture classification image/video process and fractals.
using dynamic fractal analysis,” in Proc. IEEE Int. Conf. Comput. Vis.
(ICCV), Nov. 2011, pp. 1219–1226.
[25] M. Voorons, M. Germain, G. B. Benie, and K. Fung, “Segmentation of
high resolution images based on the multifractal analysis,” in Proc. IEEE
Int. Geosci. Remote Sens. Symp., Jul. 2003, vol. 6. no. 6, pp. 3531–3533.
[26] H. Xu, J. Yan, N. Persson, W. Lin, and H. Zha, “Fractal dimension
invariant filtering and its CNN-based implementation,” in Proc. IEEE Fangxun Bao received the M.Sc. degree from the
Conf. Comput. Vis. Pattern Recognit. (CVPR), Jul. 2017, pp. 3825–3833. Department of Mathematics, Qufu Normal Univer-
[27] M. Liu, Y. Zhao, J. Liang, C. Lin, H. Bai, and C. Yao, “Depth map sity, Qufu, China, in 1994, and the Ph.D. degree
up-sampling with fractal dimension and texture-depth boundary consis- from the Department of Mathematics, Northwest
tencies,” Neurocomputing, vol. 257, no. 27, pp. 185–192, Sep. 2017. University, Xian, China, in 1997. He is currently a
[28] Y. C. Wee and H. J. Shin, “A novel fast fractal super resolu- Full Professor with the Department of Mathematics,
tion technique,” IEEE Trans. Consum. Electron., vol. 56, no. 3, Shandong University, Jinan, China. His research
pp. 1537–1541, Aug. 2010. interests include computer-aided geometric design
[29] H. Xu, G. Zhai, and X. Yang, “Single image super-resolution with and computation, computational geometry, and func-
detail enhancement based on local fractal analysis of gradient,” IEEE tion approximation.
Trans. Circuits Syst. Video Technol., vol. 23, no. 10, pp. 1740–1754,
Oct. 2013.
[30] Q. Duan, H. Zhang, Y. Zhang, and E. H. Twizell, “Bounded property
and point control of a bivariate rational interpolating surface,” Comput.
Math. Appl., vol. 52, nos. 6–7, pp. 975–984, Sep./Oct. 2006. Yifang Liu received the bachelor’s degree in
[31] Y. Zhang, F. Bao, C. Zhang, and D. Qi, “Local shape control of a electronic information science and technology and
bivariate rational interpolating surface with mixing conditions,” in Proc. the master’s degree in computer science from the
ISVD, Jun. 2011, pp. 200–205. Shandong University of Finance and Economics,
[32] Y. Zhang, F. Bao, C. Zhang, and Q. Duan, “A weighted bivari- Jinan, China, in 2012 and 2015, respectively. She
ate blending rational interpolation function and visualization control,” is currently pursuing the Ph.D. degree with the Uni-
J. Comput. Anal. Appl., vol. 14, no. 7, pp. 1303–1321, Nov. 2012. versity at Buffalo, Buffalo, NY, USA. Her research
[33] Q. Sun, F. Bao, Y. Zhang, and Q. Duan, “A bivariate rational interpola- interests include robotics, computer vision, and
tion based on scattered data on parallel lines,” J. Vis. Commun. Image image processing.
Represent., vol. 24, no. 1, pp. 75–80, Jan. 2013.
[34] S. Osher and J. A. Sethian, “Fronts propagating with curvature-
dependent speed: Algorithms based on hamilton-jacobi formulations,”
J. Comput. Phys., vol. 79, no. 1, pp. 12–49, Nov. 1988.
[35] S. Novianto, Y. Suzuki, and J. Maeda, “Near optimum estimation of Caiming Zhang received the B.S. and M.E. degrees
local fractal dimension for image segmentation,” Pattern Recognit. Lett., in computer science from Shandong University
vol. 24, nos. 1–3, pp. 365–374, Jan. 2003. in 1982 and 1984, respectively, and the Dr. Eng.
[36] M. Bevilacqua, A. Roumy, C. Guillemot, and M.-L. A. Morel, “Low- degree in computer science from the Tokyo Institute
complexity single-image super-resolution based on nonnegative neighbor of Technology, Japan, in 1994. From 1997 to 2000,
embedding,” in Proc. Brit. Mach. Vis. Conf. (BMVC), Guildford, U.K., he held a visiting position with the University of
Sep. 2012, pp. 135.1–135.10. Kentucky, Lexington, KY, USA. He is currently
[37] R. Zeyde, M. Elad, and M. Protter, “On single image scale-up using a Professor and the Doctoral Supervisor with the
sparse-representations,” in Proc. Int. Conf. Curves Surfaces, Jun. 2010, School of Computer Science and Technology, Shan-
pp. 711–730. dong University. He is also the Dean and a Professor
[38] E. Agustsson and R. Timofte, “NTIRE 2017 challenge on single image with the School of Computer Science and Technol-
super-resolution: Dataset and study,” in Proc. IEEE Conf. Comput. Vis. ogy, Shandong University of Finance and Economics. His research interests
Pattern Recognit. Workshops (CVPRW), Jul. 2017, pp. 1122–1131. include CAGD, CG, information visualization, and medical image processing.