Data with a small number of dimensions tend to be qualitatively different than

Data with a small number of dimensions tend to be

• 24

This preview shows page 11 - 15 out of 24 pages.

Data with a small number of dimensions tend to be qualitatively different than moderate or high dimensional data. In order to understand the Curse of Dimensionality , we first need to understand the other two characteristics of Data. 11 More data, more features or dimensions, is always a good thing, right? It’s actually a blessing and a curse.

Subscribe to view the full document.

Data Mining Data Data Sparsity For some datasets, most features have values of 0; in many cases fewer than 1% of the entries are non-zero. Such a data is called sparse data or it can be said that the data set has Sparsity. Can be a problem for many methods, often statistical ones. Can create a statistical bias due to small samples. Can also be an advantage, because less storage may be needed. 12
Data Mining Data Data Resolution Different resolutions reveal different patterns. If the resolution is too fine, a pattern may be buried in noise. If the resolution is too coarse, the pattern may disappear. For example, variations in atmospheric pressure on a scale of hours reflect the movement of storms and other weather systems. On a scale of months, such phenomena are not detectable. 13

Subscribe to view the full document.

Data Mining Data Curse of Dimensionality Many types of Data Analysis becomes difficult as the dimensionality (number of attributes in the data set) of the data set increases. Specifically, as dimensionality increases, the data becomes increasingly sparse in the space that it occupies . For classification, this can mean that there are not enough data objects to allow the creation of a model that reliably assigns a class to all possible objects. For clustering, the definitions of density and the distance between points, which are critical for clustering, become less meaningful.
• Winter '18
• nour

What students are saying

• As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

Kiran Temple University Fox School of Business ‘17, Course Hero Intern

• I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

Dana University of Pennsylvania ‘17, Course Hero Intern

• The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

Jill Tulane University ‘16, Course Hero Intern