G 12 hours others such as neural networks are

This preview shows page 181 - 183 out of 239 pages.

thousand features) can be fit in a reasonable amount of time (e.g., 1–2 hours).Others, such as neural networks, are feasible with datasets up to about 5,000 to10,000 features, at which point memory requirements become a limiting factor(depending upon the specific algorithm and implementation). In addition, fittingof these models can take a very long time. At the other end of the spectrum areclassifiers, such as linear discriminant analysis, that require that the number offeatures is less than the number of observations; these methods are generally usedin conjunction with some form of dimensionality reduction in order to address thislimitation.9.5.2.3 Tendency to overfitClassifiers also vary in their tendency to overfit the data. Some classifiers, such asone-nearest-neighbor classifiers and simple neural network methods, are very likelyto overfit the data, resulting in very good training performance but very bad gener-alization. However, most classification techniques use some form ofregularizationto prevent overfitting. Further, some techniques (such as support vector machines)are specifically designed to prevent overfitting.9.5.3 Which classifier is best?There is a large literature focused on the development of newer and better clas-sifier techniques, and one often sees papers in the fMRI literature that report theapplication of a novel classifier technique to fMRI data, touting its performancein comparison to other methods. It is critical to realize, however, that althoughparticular classifiers may work well for particular datasets or problems, no singlemethod is superior across all datasets and classification problems. This can in factbe proven formally and is known as the“no free lunch” theoremin machine learning(Duda et al.,2001). Each particular classifier will perform well on datasets that arewell-matched to its particular assumptions, whereas the same classifier could per-form very badly on other datasets that are mismatched. It is thus important to test arange of classifiers to ensure that the results of any study are not limited by the natureof the particular classifier being studied. Using tools such as the PyMVPA toolboxor the R software package, it is possible to assess the performance of a large panelof classifiers on any dataset, which will likely provide the best guidance as to whichclassifier is most appropriate to the problem at hand. Again here, it is important thatany such model selection is performed on data that are separate from the ultimatetest data.
1719.6 Characterizing the classifier9.5.4 Assessing classifier accuracyWe often wish to determine whether the generalization accuracy of a classifier isbetter than would be expected by chance. A simple way to assess this is by comparingperformance to that expected according to some null distribution under change(e.g., a binomial distribution for a two-alternative classification). However, if thereis any bias in the classifier, this will be inaccurate; for example, if there are unequalnumbers of observations in each class, this may introduce bias in the accuracy.

  • Left Quote Icon

    Student Picture

  • Left Quote Icon

    Student Picture

  • Left Quote Icon

    Student Picture