Coursehero >> Maryland >> UMBC >> CSEE 601

Course Hero has millions of student submitted documents similar to the one below including study guides, homework solutions, papers, exam answer keys and textbook solutions.

2003 ICML Minitutorial on Research, Riting, and Reviews Experimental Methodology Rob Holte University of Alberta holte@cs.ualberta.ca 1 February 16, 2009 Experiments serve a purpose q q q They provide evidence for claims, design them accordingly Choose appropriate test datasets, consider using artificial data Record measurements directly related to your claims 2 February 16, 2009 Establish a need q q q Try very simple approaches before complex ones Try off the shelf approaches before inventing new ones Try a wide range of alternatives not just ones most similar to yours Make sure comparisons are fair 3 February 16, 2009 q Explore limitations q Under what conditions does your system work poorly ? When does it work well ? What are the sources of variance ? q q q Eliminate as many as possible Explain the rest 4 February 16, 2009 Explore anomalies Superlinear speedup IDA* on N processors is more than N times faster than on 1 processor we were surprised to obtain superlinear speedups on average our first reaction was to assume that our sample size was too small V. Rao & V. Kumar Superlinear Speedup in Parallel State Space Search Technical Report AI88 80, 1988 CS Dept., U of Texas Austin 5 February 16, 2009 Look at your data 4 x y datasets, all with the same statistics. Are they similar ? Are they linear ? mean of the x values = 9.0 mean of the y values = 7.5 equation of the least-squared regression line is: y = 3 + 0.5x sums of squared errors (about the mean) = 110.0 regression sums of squared errors = 27.5 residual sums of squared errors (about the regression line) = 13.75 correlation coefficient = 0.82 of coefficient determination = 0.67 F.J. Anscombe (1973), "Graphs in Statistical Analysis," American Statistician, 27, 17-2 6 February 16, 2009 Anscombe datasets plotted curvepg314_Pic1 curvepg314_Pic2 7 February 16, 2009 Look at your data, again q q Japanese credit card dataset (UCI) Cross validation error rate is identical for C4.5 and 1R Is their performance the same ? 8 February 16, 2009 Closer analysis reveals Error rate is the same only on the dataset class distribution C4.5 1R ROC curves Cost curves REC curves Learning curves 9 February 16, 2009 Test alternative explanations Combinatorial auction problems CHC = hill-climbing with a clever new heuristic Solution Quality (% of optimal) problem type path match sched r75P r90P r90N arb CHC 98 99 96 83 90 89 87 10 February 16, 2009 Is CHC better than random HC ? Percentage of CHC solutions better than random HC solutions problem type % better path match sched r75P r90P r90N arb 100 100 100 63 7 6 20 11 February 16, 2009 Avoid Overtuning q Overtuning = using all your data for system development. Final system likely overfits. David Lubinksy, PhD thesis q q q Chris Drummond, PhD thesis q Held out ~10 UCI datasets for final testing Fresh data used for final testing 12 February 16, 2009 Too obvious to mention ? (no) q Debug and test your code thoroughly Keep track of parameter settings, versions of program and dataset, etc. q 13 February 16, 2009

Find millions of documents here - Study Guides, Homework Solutions, Papers, Exam Answer Keys and more. Course Hero has millions of course related materials that will enable you to learn better, faster and get an A in all your courses.
Below is a small sample set of documents: