13-mdl - Algorithmic Information Theory Consider the...

Info iconThis preview shows pages 1–5. Sign up to view the full content.

View Full Document Right Arrow Icon
Algorithmic Information Theory Consider the following Strings: 0101010101010101010101010101010101010101010101010101010101010101 1100100001100001110111101110110011111010010000100101011110010110 How much information is in each of the strings? Intuitively, very little in the first one because we can describe/compress the string as 32 01 . It seems that we can only describe the second string by its own representation, that is, the only way to describe it is by repeating its sequence of zeros and ones. Therefore, the second string seems to contain more information. We define the quantity of information contained in an object to be the size of that object’s smallest representation or description. So, | 32 01 | < | 1100100001100001110111101110110011111010010000100101011110010110 | –p .1 /1
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Minimal Description Lengths Observation: We can use algorithms to describe the structure of strings. We say that h M,w i is a description of some string x if running the ma- chine M with w as input results in string x , formally x = h M, w i Note: If we assume that we encode all our descriptions and strings as binary sequences and if w is a binary sequence, then h i = h M i w Example: A trivial description. Consider the machine I , I = "On input w , where w is a binary string: 1. accept ." That is, x = h I,x i = h I i x . –p .2/1
Background image of page 2
Minimal Description Lengths Definition: Let x be a binary string. The minimal length description of x , written d ( x ) , is the shortest string h M, w i where TM M on input w halts with x on its tape. The Kolmogorov complexity of x , K ( x ) ,is K ( x )= | d ( x ) | . Example: Let, x = 0101010101010101010101010101010101010101010101010101010101010101 , then consider the machine M 1 , M 1 = "On input w , where w is a binary string: 1. Write the string w 31 times on the tape right after the input string. 1. accept ." Now we have, x = h M 1 i 01 . If we assume that |h M 1 i| < | x |− 2 then d ( x h M 1 i 01 and therefore, K ( x | d ( x ) | = |h M 1 i 01 | –p .3/1
Background image of page 3

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Minimal Description Lengths The next theorem states that repeating a string does not significantly increase the information content of the overall string. Theorem: x q [ K ( xx ) K ( x )+ q ] . Proof: Let d ( x )= h N, w i , that is h i is a minimal lengths description of of x .Now , consider the machine M , M = "On input h T,s i , where T isaTMand s is a string: 1. Run T on s until it halts and produces a string z .
Background image of page 4
Image of page 5
This is the end of the preview. Sign up to access the rest of the document.

This note was uploaded on 10/03/2011 for the course CSC 544 taught by Professor Staff during the Spring '11 term at Rhode Island.

Page1 / 19

13-mdl - Algorithmic Information Theory Consider the...

This preview shows document pages 1 - 5. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online