Levin russ math surv 2561970 83124 192 2 algorithmic

Levin, Russ. Math. Surv. , 25:6(1970), 83–124.

192 2. Algorithmic Complexity
2.8.3. [12] Show that given x,y, and C ( x,y ), one can compute C ( x )
and C ( y ) up to an additive logarithmic term O (log C ( x,y )).
Comments. Hint: use symmetry of information and upper semicom-
putability. Suggested by L. Fortnow.
2.8.4. [28] Let ω = ω
1
ω
2
... be an infinite binary sequence. The en-
tropy function H ( p ) is defined by H ( p ) = p log 1 /p +(1 − p ) log1 / (1 − p ).
Let lim
n →∞
1
n
∑
n
i =1
1
n
ω
i
= p .
(a) Show that
C ( ω
1: n
| n ) ≤ nH
parenleftBigg
1
n
n summationdisplay
i =1
ω
i
parenrightBigg
+ log n + c.
(b) Prove the following: If the ω
i
’s are generated by coin flips with prob-
ability p for outcome 1 (a Bernoulli process with probability p ), then for
all ǫ> 0,
Pr
braceleftbigg
ω :
vextendsingle
vextendsingle
vextendsingle
vextendsingle
C ( ω
1: n
| n )
n
− H ( p )
vextendsingle
vextendsingle
vextendsingle
vextendsingle
>ǫ
bracerightbigg
→ 0 ,
as n goes to infinity.
2.8.5. [26] Show that 2 C ( a,b,c ) ≤ C ( a,b )+ C ( b,c )+ C ( c,a )+ O (log n ).
Comments. For an application relating the 3-dimensional volume of a
geometric object in Euclidean space to the 2-dimensional volumes of its
projections, see the discussion in Section 6.13 on page 530. Hint: use the
symmetry of information, Theorem 2.8.2. Source: D. Hammer and A.K.
Shen, Theor. Comput. Syst. , 31:1(1998), 1–4.
2.9
History and
References
The confluence of ideas leading to Kolmogorov complexity is analyzed
in Section 1.8 through Section 1.12. Who did what, where, and when,
is exhaustively discussed in Section 1.13. The relevant documents are
dated R.J. Solomonoff, 1960/1964, A.N. Kolmogorov, 1965, and G.J.
Chaitin, 1969. According to L.A. Levin, Kolmogorov in his talks used
to give credit also to A.M. Turing (for the universal Turing machine).
The notion of nonoptimal complexity (as a complexity based on shortest
descriptions but lacking the invariance theorem) can be attributed, in
part, also to A.A. Markov [ Soviet Math. Dokl. , 5(1964), 922–924] and
G.J. Chaitin [ J. ACM , 13(1966), 547–569], but that is not a very crucial
step from Shannon’s coding concepts.
The connection between incompressibility and randomness was made
explicit by Kolmogorov and later by Chaitin. Theorem 2.2.1 is due to
Kolmogorov. The idea to develop an algorithmic theory of information

2.9. History and References 193
is due to Kolmogorov, as is the notion of deficiency of randomness. Uni-
versal a priori probability (also based on the invariance theorem) is due
to Solomonoff. This is treated in more detail in Chapter 4. (Solomonoff
did not consider descriptional complexity itself in detail.)
In his 1965 paper, Kolmogorov mentioned the incomputability of C ( x )
in a somewhat vague form: “[ ... ] the function C
φ
( x | y ) cannot be effec-
tively calculated (generally recursive) even if it is known to be finite
for all x and y .” Also Solomonoff suggests this in his 1964 paper: “it is
clear that many of the individual terms of Eq. (1) are not ‘effectively
computable’ in the sense of Turing [ ... but can be used] as the heuristic
basis of various approximations.” Related questions were considered by
L. L¨ ofgren [ Automata Theory , E. Caianiello, ed., Academic Press, 1966,
251–268; Computer and Information Sciences II , J. Tou, ed., Academic
Press, 1967, 165–175]. Theorem 1 in the latter reference demonstrates
in general that for every universal function φ
0
, C
φ
0
( x ) is not recursive
in x . (In the invariance theorem we considered only universal functions
using a special type of coding.)
Despite the depth of the main idea of Kolmogorov complexity, the tech-
nical expression of the basic quantities turned out to be inaccurate in
the sense that many important relationships hold only to within an error