ENGRD 2700 Engineering Probability and Statistics Lecture 2: Graphical Methods; Data Summarization David S. Matteson School of Operations Research and Information Engineering Rhodes Hall, Cornell University Ithaca NY 14853 USA [e-mail] January 21, 2009 Overview Scatter &amp; <a href="/keyword/time-series/" >time series</a> Freq &amp; Hist Numerical summaries . . . Title Page Page 1 of 32 Go Back Full Screen Close Quit 1. Graphical Methods and Data Summarization Pictorial and graphical methods for data summarization. Generalities: Large amounts of data hard to interpret. Overview Reduction of the full data set to either a picture or to a small set of numerical summaries (mean, standard deviation, . . . ).. Often the full data not recoverable from the pictorial or numerical summarization; exception stem and leaf plot. Often some features not initially evident pop out immediately with the right kind of pictorial representation. Examples of techniques: 1. Stem and leaf plots 2. Simple plots: Scatter plots; <a href="/keyword/time-series/" >time series</a> plots. 3. Dot plots 4. Histograms 5. Density estimates. Scatter &amp; <a href="/keyword/time-series/" >time series</a> Freq &amp; Hist Numerical summaries . . . Title Page Page 2 of 32 Go Back Full Screen Close Quit Features we hope to nd from simple plots like the <a href="/keyword/time-series/" >time series</a> plot: Is there a trend? Is variability changing? Features we hope revealed by stem &amp; leaf, histograms or density estimates: What are the most typical or representative values? (mean &amp; median) Are there several values that might be typical (modes). Summarize the spread of the data: range and variability. Are there notable gaps in the data? Is there symmetry in the data around some typical or representative value? Are there outliers values so atypical as to be possibly measurement error or so noteworthy as to attract attention. Title Page Overview Scatter &amp; <a href="/keyword/time-series/" >time series</a> Freq &amp; Hist Numerical summaries . . . Notation Univariate case: outcomes of experiment or observational study are real numbers. Page 3 of 32 Go Back n = size of the data set; sample size x1 , x2 , . . . , xn data values. Example: observe voter behavior (R=0, D=1) and with coding record becomes 0,0,1,0,. . . ,0 (say). Full Screen Close Quit Multivariate case: outcomes of experiment or observational study are vectors of real numbers: n = size of the data set; sample size (x1 , y1 ), (x2 , y2 ), . . . , (xn , yn ) data values. Example: In an Internet study, xi = size of i-th le downloaded yi = time necessary to download the i-th le. Overview Scatter &amp; <a href="/keyword/time-series/" >time series</a> Freq &amp; Hist Numerical summaries . . . Title Page Page 4 of 32 Go Back Full Screen Close Quit 2. 2.1. Simple Plots. Scatter Plot In the plane, plot (xi , yi ) for i = 1, . . . , n; that is, put a point at abscissa xi and ordinate yi . Example: Boston University study: In 1994-5 a team of BU CS researchers collected data on student usage of the Internet. Data consists of buF = size of le requested, buL = time for the le to be downloaded, buR = inferred rate, assuming rate constant. Questions: Are large values of one variable likely to appear with large values of another? Are large les handled globally by the Internet di erently from modest sized les? Overview Scatter &amp; <a href="/keyword/time-series/" >time series</a> Freq &amp; Hist Numerical summaries . . . Title Page Page 5 of 32 Go Back Full Screen Close Quit Overview Scatter &amp; <a href="/keyword/time-series/" >time series</a> Freq &amp; Hist Numerical summaries . . . Title Page Page 6 of 32 Go Back Full Screen Close Quit Example: Exchange rate returns: For nancial data such as Pi = price at i-th measurement time, the log return is Ri = log Pi log Pi 1 , which is approximately the relative change in the price over i-th measurement period. Consider exchange rate returns relative to the US dollar for (France, Germany) (Germany, Japan) Title Page Overview Scatter &amp; <a href="/keyword/time-series/" >time series</a> Freq &amp; Hist Numerical summaries . . . Is a large movement in one currency likely to imply a large movement in another currency? Remarks: Page 7 of 32 Data is for the period prior to the introduction of the Euro. Pictures very di erent for (Fr, Ger) compared with (Ger, Jap). Go Back Full Screen Close Quit Overview Scatter &amp; <a href="/keyword/time-series/" >time series</a> Freq &amp; Hist Numerical summaries . . . Title Page AbsRet(FR) vs AbsRet(Ger) 0.06 0.05 abs(diff(log(xchr$Germany))) 0.04 Page 8 of 32 Go Back Full Screen Close Quit 0.0 0.01 0.02 0.03 0.04 0.05 0.06 abs(diff(log(xchr$France))) 0.0 0.01 0.02 0.03 Overview Scatter &amp; <a href="/keyword/time-series/" >time series</a> Freq &amp; Hist Numerical summaries . . . Title Page AbsRet(Japan) vs AbsRet(Ger) 0.06 0.05 abs(diff(log(xchr$Germany))) 0.04 Page 9 of 32 Go Back Full Screen Close Quit 0.0 0.01 0.02 0.03 0.04 0.05 0.06 abs(diff(log(xchr$Japan))) 0.0 0.01 0.02 0.03 2.2. <a href="/keyword/time-series/" >time series</a> Plots Given univariate data x1 , x2 , . . . , xn , the <a href="/keyword/time-series/" >time series</a> plot of the data is the scatter plot of the points {(i, xi ); i = 1, . . . , n}. The index i (1 i n) is sometimes called the serial index . So we plot serial index against the value of the series at that index. Choices which stat packages allow you to control to get a professional look: Overview Plotting character: What mark in the plane do you plot down at location (i, xi )? Eg: dot, heavy dot, cross, square, letter,... . What, if anything further do you do with the characters sitting in the plane? Choices typically are Leave the marks unconnected. Darken the marks and then connect them. From the mark, drop a vertical line until the x-axis. etc color axis labels to make the graph more informative. But beware of Edward Tufte s chart clutter . Scatter &amp; <a href="/keyword/time-series/" >time series</a> Freq &amp; Hist Numerical summaries . . . Title Page Page 10 of 32 Go Back Full Screen Close Quit Example: Yearly best times in the mile run: The yearly best time over 121 years in the mile run are plotted. Note an obvious trend downward which looks linear. (However, must beware over extrapolation of the linear trend, lest humans run a negative time in 2013.) Overview Scatter &amp; <a href="/keyword/time-series/" >time series</a> Freq &amp; Hist Numerical summaries . . . Title Page Page 11 of 32 Go Back Full Screen Close Quit Example: Monthly airline passenger data. International airline passengers; monthly totals in thousands of passengers from January 1949 to December 1960. Features: Obvious trend. Period of length 12 (= # months in a year) Variability increases. Overview Scatter &amp; <a href="/keyword/time-series/" >time series</a> Freq &amp; Hist Numerical summaries . . . Title Page Page 12 of 32 Go Back Full Screen Close Quit Overview Scatter &amp; <a href="/keyword/time-series/" >time series</a> Freq &amp; Hist Numerical summaries . . . Title Page Page 13 of 32 Go Back Full Screen Close Quit Example: Danish Fire Loss Data. The Danish data consist of 2492 losses in Danish Krone (DKK) from the years 1980 to 1990 inclusive. Loss=total loss for the event concerned and includes damage to buildings, damage to furniture and personal property as well as loss of pro ts. The data have been suitably adjusted to re ect 1985 values. Features: The plot is dominated by the large values. The plot needs to scale to accomodate the large values. The scaling results in obscuring features in the not-so-large values. Why worry? To convince you this might make a di erence to somebody, note that from 1970-1995, the two worst losses world wide were Hurrricane Andrew Northridge earthquake in California. Also, SF Earthquake, Exxon-Valdese oil spill, oil derrick falling over in the North Sea, . . . . Losses: In 1992 millions of dollars: Andrew $16,000 (Carl Hiasson: Stormy Weather) Northridge $11,838. Page 14 of 32 Go Back Full Screen Close Quit Overview Scatter &amp; <a href="/keyword/time-series/" >time series</a> Freq &amp; Hist Numerical summaries . . . Title Page Overview Scatter &amp; <a href="/keyword/time-series/" >time series</a> Freq &amp; Hist Numerical summaries . . . Title Page Page 15 of 32 Go Back Full Screen Close Quit 3. Frequency and Histogram Plots Compact summary; may lose some detail. Works with qualitative data sampled individual may be represented by a product name. 3.1. Dotplots Overview Scatter &amp; <a href="/keyword/time-series/" >time series</a> Freq &amp; Hist Numerical summaries . . . Read pp. 14-15 3.2. Frequency Plots and Histograms Title Page Observations on a variable can be of two types: 1. Discrete: The set S of all possible values of the variable is either nite or can be expressed as a sequence. Page 16 of 32 Go Back Full Screen Close Quit Examples: (a) Voter response: S = {0, 1}. (b) Marketing response: What brand of car will you buy? S = {Honda, Toyota, VW, Daimler-Chrysler, GM, Ford}. (c) Demand analysis: How many gallons of crude oil required each month? S = {0, 1, 2, . . . }. 2. Continuous: The set S of possible values can be a real number from a subinterval of R or even Rd . Examples: (a) When does an item fail? S = [0, ). (b) In Internet transmissions, when a packet is sent, how long until an acknowledgement ( ack ) is received? S = [0, ). (c) In economics, what is the location of maximum utility? S = R2 . Page 17 of 32 Go Back Full Screen Close Quit Overview Scatter &amp; <a href="/keyword/time-series/" >time series</a> Freq &amp; Hist Numerical summaries . . . Title Page Frequency Distribution for discrete observations. Suppose the possible values of the observations is not too large. Procedure: 1. Observe range: smallest to biggest. 2. Divide measurement axis into class intervals: disjoint intervals which cover the measurements. 3. List intervals. 4. Tally how many observations fall into an interval. 5. Compute relative number by dividing the numbers obtained in 4 by n=size of the data set. 6. Plot relative frequencies as the ordinate, with the class number as the abscissa. (If S is small, the class might be a possible value in S.) Schematically: S = {s1 , s2 , s3 }, class interval s1 s2 s3 data = {x1 , . . . , xn }. Page 18 of 32 Overview Scatter &amp; <a href="/keyword/time-series/" >time series</a> Freq &amp; Hist Numerical summaries . . . Title Page frequency # x s= s1 # x s= s2 # x s= s3 relative frequency # x s= s1 n # x s= s2 n # x s= s3 n Go Back Full Screen Close Quit Variety of ways to plot class numbers vs relative frequencies. Most popular: bar chart, histogram Just do something sensible. Overview Example: Sample 200 voters; 0=vote for McCain, 1=vote for Obama. For McCain = 89, For Obama = 111. Make a bar chart over 0,1 of heights proportional to 89/200, 111/200. Scatter &amp; <a href="/keyword/time-series/" >time series</a> Freq &amp; Hist Numerical summaries . . . Title Page Page 19 of 32 Go Back Full Screen Close Quit Histograms Simiar procedure even if data from continuous observation variable. 1. Observe range: smallest to biggest. 2. Divide measurement axis into class intervals: disjoint intervals which cover the measurements. Call these intervals I1 , . . . , Ik . Usually these are equal in length but not always. In particular, there may be an in nite interval of values bigger than a certain threshold. 3. List intervals. 4. Tally how many observations fall into an interval. Title Page Overview Scatter &amp; <a href="/keyword/time-series/" >time series</a> Freq &amp; Hist Numerical summaries . . . 5. Compute relative number by dividing the numbers obtained in 4 by n=size of the data set. Call the relative frequencies fi , i = 1, . . . k. 6. Above the ith interval, draw a rectangle centered at the midpoint of the interval such that either (a) The area is (proportional to) fi . or (b) The height is proportional to frequency. Full Screen Close Quit Page 20 of 32 Go Back Note properties of relative frequencies: 0 fi 1. k i=1 fi = 1. Therefore, the total area of the rectangles of the histogram is 1 if the area of the rectangles is relative frequency. Note: Packages will give you a choice if you want to plot Height of rectangle proportional to frequency. Rectangle area proportional to frequency. Overview Scatter &amp; <a href="/keyword/time-series/" >time series</a> Freq &amp; Hist Numerical summaries . . . Title Page Page 21 of 32 Go Back Full Screen Close Quit Example: Air passenger data. Both plottting methods give the same info when class intervals are equal. Overview Scatter &amp; <a href="/keyword/time-series/" >time series</a> Freq &amp; Hist Numerical summaries . . . Title Page Page 22 of 32 Go Back Full Screen Close Quit 3.3. How to pick classes; art vs science. The shape of the histogram varies, depending on the number and sizes of class boundaries. Usually it is desirable that the class boundaries should be chosen so that the histogram is suggestive of a smooth curve. Example: MPG. Fuel e ciency (in mile per gallon) of 19 small cars: 30.0, 32.9, 33.2, 33.6, 36.3, 36.5, 36.6, 36.7, 36.8, 36.9, 37.1, 37.2, 37.3, 37.4, 37.5, 41.0, 41.2, 42.1, 44.9. Below are four histograms of the MPG data, each drawn with a di erent number of classes. Overview Scatter &amp; <a href="/keyword/time-series/" >time series</a> Freq &amp; Hist Numerical summaries . . . Title Page Page 23 of 32 Go Back Full Screen Close Quit Overview Scatter &amp; <a href="/keyword/time-series/" >time series</a> Freq &amp; Hist Numerical summaries . . . Title Page Page 24 of 32 Go Back Full Screen Close Quit Histograms (cont): population density. hope (ponder this!) (ii) For data resulting from repeated observations on a continuous variable that there is a function f (x) satisfying (i) f (x) 0, for all x; one can (1) (2) Overview Scatter &amp; <a href="/keyword/time-series/" >time series</a> Freq &amp; Hist Numerical summaries . . . f (x)dx = 1. and one other condition. When a function f (x) satis es (1) and (2), it is called a (probability) density function. The 3rd condition we want is (iii) relative frequency of observations x x f (u)du. (3) Title Page Under proper circumstances, we think of f (x) controlling what elements are likely to be observed and the histogram is a step function approximation of the smooth function f (x). The bigger the sample size, the better hist approximates f (x). Page 25 of 32 Reassurance: This will make more sense after some probability is developed. Go Back Full Screen Close Quit Overview Scatter &amp; <a href="/keyword/time-series/" >time series</a> Freq &amp; Hist Numerical summaries . . . Title Page Page 26 of 32 Go Back Full Screen Close Quit Overview Scatter &amp; <a href="/keyword/time-series/" >time series</a> Freq &amp; Hist Numerical summaries . . . Title Page Page 27 of 32 Go Back Full Screen Close Quit Overview Scatter &amp; <a href="/keyword/time-series/" >time series</a> Freq &amp; Hist Numerical summaries . . . Title Page Page 28 of 32 Go Back Full Screen Close Quit 4. Numerical summaries of data. Measures of location: Mean &amp; median Spread of data: dispersion, sample variance (standard deviation), interquartile range. 4.1. Sample Mean Overview Scatter &amp; <a href="/keyword/time-series/" >time series</a> Freq &amp; Hist Numerical summaries . . . Sample: x1 , . . . , xn . Sample mean x := 1 n n i=1 xi =arithmetic mean. Title Page Example: 1. Suppose the sample is { a, a} for some a &gt; 0. No matter what the value of a is we have a + ( a) x= = 0. 2 The center is 0. 2. MPG of small cars: 30.0, 32.9, 33.2, 33.6, 36.3, 36.5, 36.6, 36.7, 36.8, 36.9, 37.1, 37.2, 37.3, 37.4, 37.5, 41.0, 41.2, 42.1, 44.9. mean(mpg)= 37.11579 Page 29 of 32 Go Back Full Screen Close Quit Notes: x describes the center of the sample, or a representative. x does not have to be a sample value. If the data is a, then x = 0 = a and = a. Problem with mean: quite sensitive to values which are large in absolute value. Eg: In a small town of 100 residents the max income is $80,000. Then Michael Jordon (Bill Gates?) moves to town. The new mean is no longer representative of the sample. Eg: In a class of size 10 of students, the monthly incomes are $600,. . . , $600 (nine times) and one with $1,000,000 x = 100, 540. Is this representative? Page 30 of 32 Go Back Full Screen Close Quit Overview Scatter &amp; <a href="/keyword/time-series/" >time series</a> Freq &amp; Hist Numerical summaries . . . Title Page 4.2. Sample median. Another measure of location less sensitive to extremes; ie, more robust. Data: x1 , . . . , xn ; eg, 3,2,1. Data in increasing order: x(1) x(2) x(3) ; eg 1,2,3. Sample median xn = middle value in list , average of 2 middle values , if n is odd if n is even. Overview Scatter &amp; <a href="/keyword/time-series/" >time series</a> Freq &amp; Hist Example: Recall the class of 10 students: the data were $600, $600, $600, $600, $600, $600, $600, $600, $600, $,1000,000. median= $600. Notes: In general x = xn . xn is that value such that roughly 1/2 the observations are above and 1/2 are below. Median less sensitive to extremes. Numerical summaries . . . Title Page Page 31 of 32 Go Back Full Screen Close Quit Contents Overview Scatter &amp; <a href="/keyword/time-series/" >time series</a> Freq &amp; Hist Numerical summaries . . . Title Page Page 32 of 32 Go Back Full Screen Close Quit