Lecture-accompanying prose notes

Lecture-accompanying prose notes - Partitioning in Large...

Info iconThis preview shows pages 1–2. Sign up to view the full content.

View Full Document Right Arrow Icon
Partitioning in Large Data Systems Alberto Lerner Motivation + achieving scalability through data partitioning is not new + but new desiderata: + not all systems are born large so expand as need be, just by adding nodes (disk+memory) + access patterns not known or not fixed always: so balance load as it goes + no intervention other than adding machines + Two systems have done it nicely: Bigtable (google) and Dynano (amazon) + In investigating the partitioning aspect of those, we'll study Paxos (to solve distributed consensus) and DHT's (to do away with central authorities in distributed systems). === Problems to solve To achieve. .. + . ..incremental scalability + create new partitions as data grows + accept a new node in the system + . ..adaptive load balancing + move partitions from over- to underloaded nodes + for the above to work + how to detect who is over- or underloaded in a distributed system? + how to even know who is *in* the system, if new nodes can get added? (or failures can happen) + if things get moved around how do clients find it? === Successful systems. .. + . .. had at least two distinct ways of looking at partitioning and positioning data + partition above info (no pun intended) consequently, central decisions such as balancing and group membership may need a central authority + replicate this info (roughly) everywhere so, we might get away without a central authority + First, that's Bigtable's approach. Latter, Dynamo's.
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Image of page 2
This is the end of the preview. Sign up to access the rest of the document.

This note was uploaded on 10/06/2011 for the course CS 2434 taught by Professor Shasha during the Spring '11 term at NYU.

Page1 / 3

Lecture-accompanying prose notes - Partitioning in Large...

This preview shows document pages 1 - 2. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online