132 let f def x 7 z α ω φ ω x μ dω ω α ω c 416 be the

132

* Let
F
def
= x 7→
Z
α ( ω ) φ
ω
( x ) μ ( dω ) : ∀ ω, | α ( ω ) | ≤ C (416)
be the subset of functions in the RKHS H with bounded Fourier coefficients
α ( ω ).
* Let
ˆ
F
def
=
(
x 7→
1
m
m X
i =1
α ( ω
i
) φ
ω
i
( x ) : ∀ ω, | α ( ω ) | ≤ C
)
(417)
be the subset that is spanned by the random feature functions, where ω
1: k
be
drawn i.i.d. from μ .
* Let p * be any distribution over X = R b .
* Define the inner product with respect to the data-generating distribution (this
is not the RKHS inner product):
h f, g i
def
= E
x ∼ p *
[ f ( x ) g ( x )] . (418)
* Let f * ∈ F be any true function.
* Then with probability at least 1 - δ , there exists
ˆ
f ∈
ˆ
F that
k
ˆ
f - f
*
k ≤
C
√
m
1 +
p
2 log(1 /δ ) . (419)
– Proof of Theorem 25 :
* This proof uses fairly standard tools: McDiarmid’s inequality and Jensen’s
inequality. The function we’re applying involves taking a norm of a function,
but we just need the bounded differences condition to hold.
* Fix f * ∈ F with coefficients α ( ω ).
* Construct
ˆ
f with the same coefficients, and note that
ˆ
f ∈
ˆ
F and E [
ˆ
f ] = f * .
* Define
D ( ω
1: m
) = k
ˆ
f - f
*
k . (420)
Note that D satisfies the bounded differences inequality: letting ω i
1: m
= ω
1: m
except on the i -th component, where it is ω
0
i
:
D ( ω
1: m
) - D ( ω
i
1: m
) ≤ k
ˆ
f - f
*
k - k
ˆ
f
i
- f
*
k (421)
≤ k
ˆ
f -
ˆ
f
i
k [triangle inequality] (422)
≤
1
m
k α ( ω
i
) φ
ω
i
- α ( ω
0
i
) φ
ω
0
i
k (423)
≤
2 C
m
. (424)
Note that the last line follows because | α ( ω
i
) | ≤ C and φ
ω
i
( x ) = e - i h ω
i
,x i and
| e - ia | = 1 for all a .
133

* We can bound the mean by passing to the variance:
E [ D ( ω
1: m
)] ≤
p
E [ D ( ω
1: m
) 2 ] [Jensen’s inequality] (425)
=
v
u
u
u
t E


1
m
m X
i =1
( α ( ω
i
) φ
ω
i
- f
*
)
2


[expand] (426)
=
v
u
u
t
1
m 2
m
X
i =1
E k α ( ω
i
) φ
ω
i
- f
*
k
2
[variance of i.i.d. sum]
(427)
≤
C
√
m
[use | α ( ω
i
) | ≤ C ] . (428)
* Applying McDiarmid’s inequality (Theorem 8 ), we get that
P D ( ω
1: m
) ≥
C
√
m
+ ≤ exp
- 2 2
∑
m
i =1
(2 C/m )
2
. (429)
Rearranging yields the theorem.
– Remark: the definition of α here differs from the Rahimi/Recht paper.
– Corollary:
* Suppose we had a loss function ‘ ( y, v ) which is 1-Lipschitz in the second
argument. (e.g., the hinge loss). Define the expected risk in the usual way:
L ( f )
def
= E
( x,y ) ∼ p *
[ ‘ ( y, f ( x ))] . (430)
Then the approximation ratio is bounded:
L (
ˆ
f ) - L ( f
*
) ≤ E [ | ‘ ( y,
ˆ
f ( x )) - ‘ ( y, f
*
( x )) | ] [definition, add | · | ] (431)
≤ E [ |
ˆ
f ( x ) - f
*
( x ) | ] [fix y , ‘ is Lipschitz] (432)
≤ k
ˆ
f - f
*
k [concavity of
√
· ] . (433)
– So far, we have analyzed approximation error due to having a finite m , but as-
suming an infinite amount of data. Separately, there is the estimation error due
to having n data points:
L (
ˆ
f
ERM
) - L (
ˆ
f ) ≤ O
p
C
√
n
, (434)
where
ˆ
f
ERM
minimzes the empirical risk over the random hypothesis class
ˆ
F . So,
the total error, which includes approximation error and estimation error is
L (
ˆ
f
ERM
) - L ( f
*
) = O
p
C
√
n
+
C
√
m
. (435)
134

This bound suggests that the approximation and estimation errors are balanced
when m and n are on the same order. One takeaway is that we shouldn’t over-
optimize one without the other. But one might also strongly object and say that if
m u n , then we aren’t really getting any savings! This is indeed a valid complaint,
and in order to get stronger results, we would need to impose more structure on
the problem.