This preview has intentionally blurred sections. Sign up to view the full version.
View Full DocumentThis preview has intentionally blurred sections. Sign up to view the full version.
View Full DocumentThis preview has intentionally blurred sections. Sign up to view the full version.
View Full DocumentThis preview has intentionally blurred sections. Sign up to view the full version.
View Full DocumentThis preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
Unformatted text preview: 1 Data preparation 2 Â¡ Data imputation Â¡ Data transformation Â¡ Data sampling 3 Data Imputation Â¡ Data imputation is to replace missing value by some other nonmissing values Â¡ During data cleaning, missing values are identified. Due to the nature of some data analysis models, observations with missing values in one or more variables might not be useful Â¢ A multiple regression model requires that no missing values in any explanatory variables Â¢ A principal component analysis requires no missing values in the data 4 Treating missing values Â¡ Ignore those observations with missing values Â¢ Not very effective â€“ the effective number of observations is reduced Â¢ Especially poor for data analysis with many variables â€“ consider 20 variables, each with 2% missing data, assume independent, only (10.02) 20 = 66.8% observations without missing values 5 Â¡ Use some values to fill in the missing value â€“ imputation Â¢ A global constant â€“ replace all missing values by a single constant, e.g. â€˜missingâ€™, â€˜unknownâ€™, or a specific numeric code Â¢ The variable mean â€“ replace all missing values by the mean of the variable (or other summary statistics, e.g. median, mode, etc.) Â¢ The variable mean for all samples belonging to the same class as the observation with the missing value, i.e. divide population into subpopulations before replacing missing values by the mean 6 Â¡ Use the most probable values to fill in the missing value Â¢ This is a more advanced method Â¢ Determined by statistical modeling tools such as regression, inferencebased tools, decision tree etc. Â¡ Do nothing Â¢ Some data analysis methods allow missing values. E.g. decision tree. Observations with missing data may be used in these methods Â¢ This may be the most natural way to deal with missing data for these methods 7 Â¡ Advantages Â¢ More effective number of observations Â¢ Some data analysis methods do not allow missing values Â¡ Disadvantages Â¢ Introduce more uncertainty to the data Â¢ Will the imputed values be biased? Â¢ Will they distort the original data? 8 Example Â¡ Data set LOAN contains information about customers who apply for a bank loan. Customerâ€™s income is among the information Â¡ There are 10 out of 700 customers having missing values for customerâ€™s income Â¡ A simple way to impute the income for these customers is to use the sample mean 9 * example 11.1  impute missing value by sample mean; proc means data=ch11.loan noprint; var income; output out=sumfile mean=meanincome; run; data newloan; set ch11.loan; if _n_=1 then set sumfile (keep=meanincome); if income= . then income=meanincome; drop meanincome; run; proc print data=newloan; run; 10 Â¡ The sample mean of INCOME is output to the summary data set SUMFILE Â¡ Merge the summary data set SUMFILE (contains one observation only) with the original LOAN data set....
View
Full
Document
This note was uploaded on 02/09/2012 for the course STAT 1301 taught by Professor Smslee during the Spring '08 term at HKU.
 Spring '08
 SMSLee
 Statistics

Click to edit the document details