Efficient Model Selection for Sparse Least-Square SVMs

Algorithm 1

The Reduced Forward Least-Squares Approximation SVM.

INPUT:

 (i) The data set {(𝐱₁,𝑦₁),…,(𝐱_ℓ,𝑦_ℓ)}

 (ii) 𝑚 which is the number of support vectors desired in the expansion of the solution and 1≤𝑚≤ℓ

 (iii) A dictionary of 𝑚 basis functions 𝐃₀={𝐝₁,…,𝐝_𝑚}

INITIALIZATION:

 (i) Generate a permutation of integers between 1 and ℓ. The first 𝑚 elements form a vector Γ={𝛾₁,…,𝛾_𝑚}

    which are the indices of randomly-sampled columns from the dictionary matrix 𝐃₀.

 (ii) Current residue vector 𝐲, current dictionary 𝐃 which is initially a matrix of evaluations of ℓ candidate basis functions on

     training data:

             𝑦𝐲⟵⋮¹𝑦_ℓ𝑑and𝐃⟵^𝛾1(𝐱₁)…𝑑_𝛾𝑚(𝐱₁)𝑑⋮⋱⋮_𝛾1(𝐱_ℓ)…𝑑_𝛾𝑚(𝐱_ℓ)

 (iii) The matrix 𝐀 and the vector b both starts as empty 𝐀 is appended a row and 𝐛 grows by one extra element

      at each iteration, which in the end forms a linear system.

 (iv) A variable 𝑝 which is the pointer to the current investigated basis functions and also a count of selected basis functions. At

   the start, 𝑝=0.

FOR  𝑖=1,…,𝑚 AND 𝛾_𝑖≠−1

  𝑝←𝑝+1

             𝑏_𝑝⟵𝐃.,𝛾_𝑖^𝑇𝐲𝐃.,𝛾_𝑖²

 (v) The residue vector is reduced by 𝑏_𝑝𝐝_𝛾𝑖 as the target values for the next linear system of size 𝑙:

                    𝐲⟵𝐲−𝑏_𝑝𝐃(.,𝛾_𝑖)

 (vi) Update the dictionary matrix and prune the candidate basis functions which can be represented as a linear combinations

     of the previously selected ones:

   FOR  𝑗=𝑖+1,…,𝑚 AND 𝛾_𝑗≠−1

                   𝛽_𝛾𝑗⟵𝐃.,𝛾_𝑖^𝑇𝐃.,𝛾_𝑗𝐃.,𝛾_𝑖²

         𝐃.,𝛾_𝑖⟵𝐃.,𝛾_𝑗−𝛽_𝑗𝐃.,𝛾_𝑖

    IF  𝐃.,𝛾_𝑗=𝟎

         𝛾_𝑖⟵−1

 𝛽_𝛾𝑖⟵1

 𝐃.,𝛾_𝑗⟵𝟎

 𝛽𝐀⟵¹,…,𝛽_𝑚𝐀

 𝑏𝐛⟵^𝑝𝐛

BACK SUBSTITUTION:

 The 𝑝 positive elements of Γ, which is represented by Λ={𝜆₁,…,𝜆_𝑝} in ascending order, are the indices of the 𝑝 selected basis

 functions. 𝑝 columns of matrix 𝐀 whose indices are Λ and 𝐛 forms a linear system, on which the process of back substitution is

 performed for the solution:

          𝛼_𝑝⟵𝑏_𝑝

   FOR  𝑖=𝑝−1,…,1

                    𝛼_𝑖⟵𝑏_𝑖−∑^𝑝_𝑗=𝑖+1𝛼_𝑗𝐀(𝑖,𝜆_𝑗)

OUTPUT:

 The solution is defined by 𝑓(𝐱)=∑^𝑝_𝑖=1𝛼_𝑖𝐝_𝜆𝑖(𝐱)