DWDMUNIT8[ajntuworld.in] - UNIT-8 Mining Complex Types of...

Info icon This preview shows page 1. Sign up to view the full content.

View Full Document Right Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: UNIT-8 Mining Complex Types of Data Lecture Topic ************************************************** Lecture-50 Multidimensional analysis and descriptive mining of complex data objects Lecture-51 Mining spatial databases Lecture-52 Mining multimedia databases Lecture-53 Mining time-series and sequence Lecture-54 Mining text databases Lecture-55 Mining the World-Wide Web data Lecture-50 Multidimensional analysis and descriptive mining of complex data objects Mining Complex Data Objects: Generalization of Structured Data Set-valued attribute Generalization of each value in the set into its corresponding higher-level concepts Derivation of the general behavior of the set, such as the number of elements in the set, the types or value ranges in the set, or the weighted average for numerical data hobby = {tennis, hockey, chess, violin, nintendo_games} generalizes to {sports, music, video_games} List-valued or a sequence-valued attribute Same as set-valued attributes except that the order of the elements in the sequence should be observed in the generalization Lecture-50 - Multidimensional analysis and descriptive mining of complex data objects Generalizing Spatial and Multimedia Data Spatial data: Generalize detailed geographic points into clustered regions, such as business, residential, industrial, or agricultural areas, according to land usage Require the merge of a set of geographic areas by spatial operations Image data: Extracted by aggregation and/or approximation Size, color, shape, texture, orientation, and relative positions and structures of the contained objects or regions in the image Music data: Summarize its melody: based on the approximate patterns that repeatedly occur in the segment Summarized its style: based on its tone, tempo, or the major musical instruments played Lecture-50 - Multidimensional analysis and descriptive mining of complex data objects Generalizing Object Data Object identifier: generalize to the lowest level of class in the class/subclass hierarchies Class composition hierarchies generalize nested structured data generalize only objects closely related in semantics to the current one Construction and mining of object cubes Extend the attribute-oriented induction method Apply a sequence of class-based generalization operators on different attributes Continue until getting a small number of generalized objects that can be summarized as a concise in high-level terms For efficient implementation Examine each attribute, generalize it to simple-valued data Construct a multidimensional data cube (object cube) Problem: it is not always desirable to generalize a set of values to single-valued data Lecture-50 - Multidimensional analysis and descriptive mining of complex data objects An Example: Plan Mining by Divide and Conquer Plan: a variable sequence of actions E.g., Travel (flight): <traveler, departure, arrival, d-time, a-time, airline, price, seat> Plan mining: extraction of important or significant generalized (sequential) patterns from a planbase (a large collection of plans) E.g., Discover travel patterns in an air flight database, or find significant patterns from the sequences of actions in the repair of automobiles Method Attribute-oriented induction on sequence data A generalized travel plan: <small-big*-small> Divide & conquer:Mine characteristics for each subsequence E.g., big*: same airline, small-big: nearby region Lecture-50 - Multidimensional analysis and descriptive mining of complex data objects A Travel Database for Plan Mining Example: Mining a travel planbase Travel plans table plan# 1 1 1 1 2 . . . action# 1 2 3 4 1 . . . departure ALB JFK ORD LAX SPI . . . depart_time 800 1000 1300 1710 900 . . . arrival JFK ORD LAX SAN ORD . . . arrival_time 900 1230 1600 1800 950 . . . airline TWA UA UA DAL AA . . . … … … … … … . . . Airport info table airport_code 1 1 1 1 2 . . . city 1 2 3 4 1 . . . state ALB JFK ORD LAX SPI . . . region airport_size 800 1000 1300 1710 900 . . . … … … … … … . . . Lecture-50 - Multidimensional analysis and descriptive mining of complex data objects Multidimensional Analysis Strategy A multi-D model for the planbase Generalize the planbase in different directions Look for sequential patterns in the generalized plans Derive high-level plans Lecture-50 - Multidimensional analysis and descriptive mining of complex data objects Multidimensional Generalization Multi-D generalization of the planbase Plan# 1 2 Loc_Seq ALB - JFK - ORD - LAX - SAN SPI - ORD - JFK - SYR . . . . . . Size_Seq S-L-L-L-S S-L-L-S State_Seq N-N-I-C-C I-I-N-N . . . Merging consecutive, identical actions in plans Plan# 1 2 . . . Size_Seq S - L+ - S S - L+ - S State_Seq N+ - I - C+ I+ - N+ . . . Region_Seq E+ - M - P+ M+ - E+ … … … . . . flight ( x, y, ) airport _ size ( x, S ) airport _ size( y, L) region ( x) region ( y ) [75%] Lecture-50 - Multidimensional analysis and descriptive mining of complex data objects Generalization-Based Sequence Mining Generalize planbase in multidimensional way using dimension tables Use no of distinct values (cardinality) at each level to determine the right level of generalization (level-“planning”) Use operators merge “+”, option “” to further generalize patterns Retain patterns with significant support Lecture-50 - Multidimensional analysis and descriptive mining of complex data objects Generalized Sequence Patterns AirportSize-sequence survives the min threshold (after applying merge operator): S-L+-S [35%], L+-S [30%], S-L+ [24.5%], L+ [9%] After applying option operator: [S]-L+-[S] [98.5%] Most of the time, people fly via large airports to get to final destination Other plans: 1.5% of chances, there are other patterns: S-S, L-S-L Lecture-50 - Multidimensional analysis and descriptive mining of complex data objects Lecture-51 Mining spatial databases Spatial Data Warehousing Spatial data warehouse: Integrated, subject-oriented, time-variant, and nonvolatile spatial data repository for data analysis and decision making Spatial data integration: a big issue Structure-specific formats (raster- vs. vector-based, OO vs. relational models, different storage and indexing) Vendor-specific formats (ESRI, MapInfo, Integraph) Spatial data cube: multidimensional spatial database Both dimensions and measures may contain spatial components Lecture-51 - Mining spatial databases Dimensions and Measures in Spatial Data Warehouse Dimension modeling nonspatial e.g. temperature: 25-30 degrees generalizes to hot spatial-to-nonspatial e.g. region “B.C.” generalizes to description “western provinces” spatial-to-spatial e.g. region “Burnaby” generalizes to region “Lower Mainland” Measures numerical distributive ( count, sum) algebraic (e.g. average) holistic (e.g. median, rank) spatial collection of spatial pointers (e.g. pointers to all regions with 25-30 degrees in July) Lecture-51 - Mining spatial databases Input Example: BC weather pattern analysis A map with about 3,000 weather probes scattered in B.C. Daily data for temperature, precipitation, wind velocity, etc. Concept hierarchies for all attributes Output A map that reveals patterns: merged (similar) regions Goals Interactive analysis (drill-down, slice, dice, pivot, roll-up) Fast response time Minimizing storage space used Challenge A merged region may contain hundreds of “primitive” regions (polygons) Lecture-51 - Mining spatial databases Star Schema of the BC Weather Warehouse Spatial data warehouse Dimensions region_name time temperature precipitation Measurements region_map area count Dimension table Lecture-51 - Mining spatial databases Fact table Spatial Merge Precomputing all: too much storage space On-line merge: very expensive Lecture-51 - Mining spatial databases Methods for Computation of Spatial Data Cube On-line aggregation: collect and store pointers to spatial objects in a spatial data cube expensive and slow, need efficient aggregation techniques Precompute and store all the possible combinations huge space overhead Precompute and store rough approximations in a spatial data cube accuracy trade-off Selective computation: only materialize those which will be accessed frequently a reasonable choice Lecture-51 - Mining spatial databases Spatial Association Analysis Spatial association rule: A B [s%, c%] A and B are sets of spatial or nonspatial predicates Topological relations: intersects, overlaps, disjoint, etc. Spatial orientations: left_of, west_of, under, etc. Distance information: close_to, within_distance, etc. s% is the support and c% is the confidence of the rule Examples is_a(x, large_town) ^ intersect(x, highway) adjacent_to(x, water) [7%, 85%] is_a(x, large_town) ^adjacent_to(x, georgia_strait) close_to(x, u.s.a.) [1%, 78%] Lecture-51 - Mining spatial databases Progressive Refinement Mining of Spatial Association Rules Hierarchy of spatial relationship: g_close_to: near_by, touch, intersect, contain, etc. First search for rough relationship and then refine it Two-step mining of spatial association: Step 1: Rough spatial computation (as a filter) Using MBR or R-tree for rough estimation Step2: Detailed spatial algorithm (as refinement) Apply only to those objects which have passed the rough spatial association test (no less than min_support) Lecture-51 - Mining spatial databases Spatial Classification and Spatial Trend Analysis Spatial classification Analyze spatial objects to derive classification schemes, such as decision trees in relevance to certain spatial properties (district, highway, river, etc.) Example: Classify regions in a province into rich vs. poor according to the average family income Spatial trend analysis Detect changes and trends along a spatial dimension Study the trend of nonspatial or spatial data changing with space Example: Observe the trend of changes of the climate or vegetation with the increasing distance from an ocean Lecture-51 - Mining spatial databases Lecture-52 Mining multimedia databases Similarity Search in Multimedia Data Description-based retrieval systems Build indices and perform object retrieval based on image descriptions, such as keywords, captions, size, and time of creation Labor-intensive if performed manually Results are typically of poor quality if automated Content-based retrieval systems Support retrieval based on the image content, such as color histogram, texture, shape, objects, and wavelet transforms Lecture-52 - Mining multimedia databases Queries in Content-Based Retrieval Systems Image sample-based queries: Find all of the images that are similar to the given image sample Compare the feature vector (signature) extracted from the sample with the feature vectors of images that have already been extracted and indexed in the image database Image feature specification queries: Specify or sketch image features like color, texture, or shape, which are translated into a feature vector Match the feature vector with the feature vectors of the images in the database Lecture-52 - Mining multimedia databases Approaches Based on Image Signature Color histogram-based signature The signature includes color histograms based on color composition of an image regardless of its scale or orientation No information about shape, location, or texture Two images with similar color composition may contain very different shapes or textures, and thus could be completely unrelated in semantics Multifeature composed signature The signature includes a composition of multiple features: color histogram, shape, location, and texture Can be used to search for similar images Lecture-52 - Mining multimedia databases Wavelet Analysis Wavelet-based signature Use the dominant wavelet coefficients of an image as its signature Wavelets capture shape, texture, and location information in a single unified framework Improved efficiency and reduced the need for providing multiple search primitives May fail to identify images containing similar in location or size objects Wavelet-based signature with region-based granularity Similar images may contain similar regions, but a region in one image could be a translation or scaling of a matching region in the other Compute and compare signatures at the granularity of regions, not the entire image Lecture-52 - Mining multimedia databases C-BIRD: Content-Based Image Retrieval from Digital libraries Search by image colors by color percentage by color layout by texture density by texture Layout by object model by illumination invariance by keywords Lecture-52 - Mining multimedia databases Multi-Dimensional Search in Color layout Multimedia Databases Lecture-52 - Mining multimedia databases Multi-Dimensional Analysis in Multimedia Databases Color histogram Texture layout Lecture-52 - Mining multimedia databases Mining Multimedia Databases Refining or combining searches Search for “airplane in blue sky” (top layout grid is blue and keyword = “airplane”) Search for “blue sky” (top layout grid is blue) Search for “blue sky and green meadows” (top layout grid is blue and bottom is green) Lecture-52 - Mining multimedia databases Multidimensional Analysis of Multimedia Data Multimedia data cube Design and construction similar to that of traditional data cubes from relational data Contain additional dimensions and measures for multimedia information, such as color, texture, and shape The database does not store images but their descriptors Feature descriptor: a set of vectors for each visual characteristic Color vector: contains the color histogram MFC (Most Frequent Color) vector: five color centroids MFO (Most Frequent Orientation) vector: five edge orientation centroids Layout descriptor: contains a color layout vector and an edge layout vector Lecture-52 - Mining multimedia databases Mining Multimedia Databases in Lecture-52 - Mining multimedia databases Mining Multimedia Databases The Data Cube and the Sub-Space Measurements G JP EG IF all Sm dium e Me arge y Larg L er V By Size By Format By Format & Size RED WHITE BLUE Cross Tab JPEG GIF By Colour RED WHITE BLUE Group By Colour RED WHITE BLUE Measurement Sum By Colour & Size Sum By Format Sum By Format & Colour By Colour • Format of image • Duration • Colors • Textures • Keywords • Size • Width • Height • Internet domain of image • Internet domain of parent pages • Image popularity Lecture-52 - Mining multimedia databases Classification in MultiMediaMiner Lecture-52 - Mining multimedia databases Mining Associations in Multimedia Data Special features: Need # of occurrences besides Boolean existence, e.g., “Two red square and one blue circle” implies theme “air-show” Need spatial relationships Blue on top of white squared object is associated with brown bottom Need multi-resolution and progressive refinement mining It is expensive to explore detailed associations among objects at high resolution It is crucial to ensure the completeness of search at multi-resolution space Lecture-52 - Mining multimedia databases Mining Multimedia Databases Spatial Relationships from Layout property P1 on-top-of property P2 property P1 next-to property P2 Different Resolution Hierarchy Lecture-52 - Mining multimedia databases Mining Multimedia Databases From Coarse to Fine Resolution Mining Lecture-52 - Mining multimedia databases Challenge: Curse of Dimensionality Difficult to implement a data cube efficiently given a large number of dimensions, especially serious in the case of multimedia data cubes Many of these attributes are set-oriented instead of single-valued Restricting number of dimensions may lead to the modeling of an image at a rather rough, limited, and imprecise scale More research is needed to strike a balance between efficiency and power of representation Lecture-52 - Mining multimedia databases Lecture-53 Mining time-series and sequence data Mining Time-Series and Sequence Data Time-series database Consists of sequences of values or events changing with time Data is recorded at regular intervals Characteristic time-series components Trend, cycle, seasonal, irregular Applications Financial: stock price, inflation Biomedical: blood pressure Meteorological: precipitation Lecture-53 - Mining time-series and sequence data Mining Time-Series and Sequence Data Time-series plot Lecture-53 - Mining time-series and sequence data Mining Time-Series and Sequence Data: Trend analysis A time series can be illustrated as a time-series graph which describes a point moving with the passage of time Categories of Time-Series Movements Long-term or trend movements (trend curve) Cyclic movements or cycle variations, e.g., business cycles Seasonal movements or seasonal variations i.e, almost identical patterns that a time series appears to follow during corresponding months of successive years. Irregular or random movements Lecture-53 - Mining time-series and sequence data Estimation of Trend Curve The freehand method Fit the curve by looking at the graph Costly and barely reliable for large-scaled data mining The least-square method Find the curve minimizing the sum of the squares of the deviation of points on the curve from the corresponding data points The moving-average method Eliminate cyclic, seasonal and irregular patterns Loss of end data Sensitive to outliers Lecture-53 - Mining time-series and sequence data Discovery of Trend in Time-Series Estimation of seasonal variations Seasonal index Set of numbers showing the relative values of a variable during the months of the year E.g., if the sales during October, November, and December are 80%, 120%, and 140% of the average monthly sales for the whole year, respectively, then 80, 120, and 140 are seasonal index numbers for these months Deseasonalized data Data adjusted for seasonal variations E.g., divide the original monthly data by the seasonal index numbers for the corresponding months Lecture-53 - Mining time-series and sequence data Discovery of Trend in Time-Series Estimation of cyclic variations If (approximate) periodicity of cycles occurs, cyclic index can be constructed in much the same manner as seasonal indexes Estimation of irregular variations By adjusting the data for trend, seasonal and cyclic variations With the systematic analysis of the trend, cyclic, seasonal, and irregular components, it is possible to make long- or short-term predictions with reasonable quality Lecture-53 - Mining time-series and sequence data Similarity Search in Time-Series Analysis Normal database query finds exact match Similarity search finds data sequences that differ only slightly from the given query sequence Two categories of similarity queries Whole matching: find a sequence that is similar to the query sequence Subsequence matching: find all pairs of similar sequences Typical Applications Financial market Market basket data analysis Scientific databases Medical diagnosis Lecture-53 - Mining time-series and sequence data Multidimensional Indexing Multidimensional index Constructed for efficient accessing using the first few Fourier coefficients Use the index can to retrieve the sequences that are at most a certain small distance away from the query sequence Perform postprocessing by computing the actual distance between sequences in the time domain and discard any false matches Lecture-53 - Mining time-series and sequence data Subsequence Matching Break each sequence into a set of pieces of window with length w Extract the features of the subsequence inside the window Map each sequence to a “trail” in the feature space Divide the trail of each sequence into “subtrails” and represent each of them with minimum bounding rectangle Use a multipiece assembly algorithm to search for longer sequence matches Lecture-53 - Mining time-series and sequence data Enhanced similarity search methods Allow for gaps within a sequence or differences in offsets or amplitudes Normalize sequences with amplitude scaling and offset translation Two subsequences are considered similar if one lies within an envelope of width around the other, ignoring outlie...
View Full Document

{[ snackBarMessage ]}

What students are saying

  • Left Quote Icon

    As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

    Student Picture

    Kiran Temple University Fox School of Business ‘17, Course Hero Intern

  • Left Quote Icon

    I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

    Student Picture

    Dana University of Pennsylvania ‘17, Course Hero Intern

  • Left Quote Icon

    The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

    Student Picture

    Jill Tulane University ‘16, Course Hero Intern