EE 376B/Stat 376B
Handout #19
Information Theory
Tuesday, May 30, 2006
Prof. T. Cover
Solutions to Homework Set #7
1.
Images.
Consider an
n
×
n
array
x
of 0’s and 1’s . Thus
x
has
n
2
bits.
(
a
)
(
b
)
(
c
)
Find the Kolmogorov complexity
K
(
x

n
) (to ﬁrst order) if
(a)
x
is a horizontal line.
(b)
x
isasquare
.
(c)
x
is the union of two lines, each line being vertical or horizontal.
(d)
x
is a random array.
(e)
x
is a rectangle lined up with the axes.
(f)
x
is the union of two such rectangles meeting in a corner.
Solution: Images.
(a) The program to print out an image of one horizontal line is of the form
For
1
≤
i
≤
n
{
Set pixels on row
i
to 0;
}
Set pixels on row
r
to 1;
Print out image.
Since the computer already knows
n
, the length of this program is
K
(
r

n
)+
c
,
which is
≤
log
n
+
c
. Hence, the Kolmogorov complexity of a line image is
K
(line

n
)
≤
log
n
+
c.
(1)
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document(b) For a square, we have to tell the program the coordinates of the top left corner,
and the length of the side of the square. This requires no more than 3 log
n
bits,
and hence
K
(square

n
)
≤
3log
n
+
c.
However, we can save some description length by ﬁrst describing the length of the
side of the square and then the coordinates. Knowing the length of the side of
the square reduces the range of possible values of the coordinates. Even better,
we can count the total number of such squares. There is one
n
×
n
square, four
(
n

1)
×
(
n

1) squares, nine (
n

2)
×
(
n

2) squares, etc. The total number
of squares is
1
2
+2
2
+3
2
+
···
+
n
2
=
n
(
n
+ 1)(2
n
+1)
6
≈
n
3
3
.
Since we can give the index of a square in a lexicographic ordering,
This is the end of the preview.
Sign up
to
access the rest of the document.
 Spring '05
 TomCover
 arbitrary sequence

Click to edit the document details