Dimm's
Reference: FB "Memory Systems: Cache, DRAM, Disk Bruce Jacob, Spencer Ng, & David Wang Today's material & any uncredited diagram came from Chapter 14
School of Computing University of Utah
1
CS7810
The Problem
Multi-drop busses don't scale well
demand for higher memory bandwidth continues traditional memory architecture scaling
can happen in speed or capacity but not both market expectation
more for the same price OR more for less
Intel's idea but solution is a common one
replace with point to point signaling
mitigates signal integrity problem introduces the multi-hop problem
Short story
already has proven to be problematic Intel now moving to BoB (Buffer on Board)
essentially a multi-spigot FB Dimm idea
School of Computing University of Utah
2
CS7810
Page 1
ITRS Comparison Humor
2001 ITRS
Remember in 2001: frequency wars were still ongoing, and power worries were just starting to peek over the horizon
Process (nm) CPU GHz Mlogic T/CM2 Hperf pin ct. HP cents/pin Mem cents/pin Mem pin ct. CPU cost/pkg Max Mem cost/ pkg Min Mem cost/ pkg
2004 90 3.99 77.2 2263 1.88 0.34-1.39 48-160 42.5444 2.224 0.1632
2007 65 6.74 154.3 3012 1.61 0.27-0.84 48-160 48.4932 1.344 0.1296
2010 45 12 309 4009 1.68 0.22-0.34 62-208 67.3512 0.7072 0.1364
2013 32 19 617 5335 1.44 0.19-0.39 81-270 76.824 1.053 0.1539
2016 22 29 1235 7100 1.22 0.19-0.33 105-351 86.62 1.1583 0.1995
2008 ITRS
Process (nm) CPU GHz 12 inv delays 2007 68 4.7 154 3072 .69-1.13 .27-.5 2010 45 5.8 309 3072 .60-1.20 .23-.44 2013 32 7.3 617 3072 .51-.87 .20-.38 2016 22.5 9.1 1235 3072 .44-.75 .20-.32
Mlogic T/CM2 Hperf pin ct. 33% P & G HP cents/pin Mem cents/pin Mem pin ct. CPU cost/pkg Max Mem cost/ pkg Min Mem cost/ pkg
Unspecified in 2008 ITRS Update
Note in 2008: lack of specification on memory packaging and cost no frequency prediction other than 12-inverter delay
School of Computing University of Utah
3
CS7810
FB Dimm Idea
Need to create higher bandwidth
DDR2
400 MT/s configured
up to 4 two-rank DIMM's
DDR3
800 MT/s configured
1 2-rank Dimm
Move multi-drop bus to the DIMM
daisy chain the DIMMs using an AMB ASIC
AMB ::= Advanced Memory Buffer
actually much more than a buffer also does bit-lane retiming packetized frame-relay protocol
AMD duties
extract DRAM commands from frame control DRAM devices (2ndary mem.ctlr)
School of Computing University of Utah
4
CS7810
Page 2
Replacement Strategy
THIS
WITH THIS
School of Computing University of Utah
5
CS7810
FB Dimm Problems
Device compatibility (the usual boat anchor)
still need
JEDEC standard adopted after Intel push use commodity DDR2&3 components retain user configuration flexibility
However
significant increase in idle system latency increased power consumption
big problem
AMB cost adder
incompatible with DRAM market economics
Result
lots of resistance from system manufacturers
remember Intel makes the parts not the system
School of Computing University of Utah
6
CS7810
Page 3
The AMB ASIC
One hop in a daisy chain
role (incoming side = southbound channel)
examine frame contents is it for me?
if so broadcast to the DRAM's on the appropriate rank if not recondition signals and pass it on
marshal write data to the DRAM's read data converted into frames to place on northbound channel classic store and forward network with a wrinkle wrinkle is essentially cut-through routing forward before digest and check reduces latency but increases affected scope of errors
role (outbound = northbound side)
encapsulate data burst into frames
this involves a significant amount of bit-lane retiming
Benefit
6x FBD signaling rate over DRAM devices for DDR2
School of Computing University of Utah
7
CS7810
2 Rank FB-Dimm Diagram
FBD clock rate = 4 GT/s
DDR2 clock rate = 667 MT/s
School of Computing University of Utah
8
CS7810
Page 4
FB Dimm Mesochronous Timing
Multiple clock domains
synchronous w.r.t. each other BUT phase relationships are not strictly defined
hence the need for bit-lane retiming
skew and jitter exceed bit cycle times removes need for trace length equality
"stub electronics" problem a major stumbling block in non-FB DDR2 memory systems simplifies interconnect design at the expense of active "correct it" silicon
north- and south-bound lanes designed to be timing independent
As always
the devil is in the details
so let's look at some of them
School of Computing University of Utah
9
CS7810
Signaling & Timing
Not all that different borrowed technology
Northbridge
likely contains both the PCIe and Mem_Ctlr so use PCIe style signaling
well understood technology 1.5v differential signaling
optimized for FR4 PCB's
School of Computing University of Utah
10
CS7810
Page 5
Clock Data Recovery
Common problem
clock doesn't have known phase relationship with data
one known technique
recover clock from the data signal but this requires a known number of signal transitions real data doesn't look this way so encoding is required
8b/10b Fibre Channel or HyperTransport scrambling models for example provided DC balance electrically important simplifies clock recovery by insuring that enough transitions occur per some unit of time
result
use transitions to recover clock use recovered clock to determine data
implied: clock skew+jitter doesn't change wildly in short time frame
Actual FB-Dimm standard uses a simpler approach
no inter-lane phase relationships specified
does specify transition density
6 transition minimum in a 512 bit frame
School of Computing University of Utah
11
CS7810
Unit Interval
DDR & 6:1
12 UI's/DRAM clock cycle form the FB-Dimm "frame"
Bit lane independence
cause: latency and path length variations result: several UI difference in lane burst arrival at an AMB
FB-Dimm and AMB requirements
logic to deskew the data across the independent bit lanes danger: increased latency = de-skew-time*hop-count
School of Computing University of Utah
12
CS7810
Page 6
Benefit: Less Routing Restriction
Source: Intel
School of Computing University of Utah
13
CS7810
Resync Latency Cost
Forwarding delay dominated by slowest lane
Too slow if resync is done on every hop
hence 2 southbound frame relay modes
resample
clock recovery removes bit jitter in a lane does not correct lane UI skew
spec allows a maximum of 46 UI difference between lanes
resync
delay retransmit until all lanes are collected then drive resynchronized frame
School of Computing University of Utah
14
CS7810
Page 7
3 AMB Datapaths
Resample and Resync Plus need to extract southbound command
in case target is this DIMM note forward anyway style
decode and forward if it's not for me option is intractable
since decode time would have to be added to each southbound hop
School of Computing University of Utah
15
CS7810
Protocol
Asymmetric channels
southbound
10 bit lanes * 12 UI's = 120 bits/frame
half peak write bandwidth 4 UI's for command hence 80 write data bits/frame
northbound
14 bit lanes * 12 UI's = 168 bits/frame
full peak read bandwidth of a target rank
both contain CRC info for data recovery at receiver
and actual data/frame is less: 72 (64+8) & 144 (128+16)
to support fail over mechanism (more on this soon)
3 common frame types
School of Computing University of Utah
16
CS7810
Page 8
Frame Formats
Southbound command only
3 commands/frame
sent to independent DIMMs or ranks
improves parallelism can also allow certain modules to be moved to a lower power state
nops or platform specific debug patterns pad frame when 3 commands aren't needed
Southbound command and write data
command, 64 data, and 8 check bits
8 bits can be used as a byte mask if DIMM doesn't support ECC
weirdness
multiple frames are needed for a full write burst
they do not need to be contiguous (indicates read priority model)
each write-data subframe only contains 1 bit of the target AMB address
3 subframes needed to form full address (8 DIMM max spec) implies ALL AMB's must buffer data write before destination is known energy cost of writes exacerbated
Northbound read 1 DIMM cycle read return 128 + 16
School of Computing University of Utah
17
CS7810
Commands
2 types
channel
manage the AMB's
debug read and write configuration register clock enable management soft channel reset
recover when a transmission error is detected mem_ctlr detects CRC error or AMB signals via an alert frame reset and then retry all writes that weren't committed insure that AMB clock recovery circuits see the min. # of transitions southbound transitions provided by mem_ctlr as fake write data northbound response last DIMM sends fake read return must be inserted once every 42-46 frames (JEDEC standard) implies channel can't be powered down easily (another power defect)
channel sync
DRAM
AMB's decode and send to DRAM devices on the DIMM
School of Computing University of Utah
18
CS7810
Page 9
Frame and Command Scheduling
Interesting set of choices
master to multiple slave controllers (obvious) FB mem_ctlr still maintains total control of:
DRAM and frame scheduling
minimizes logic in AMB's AMB's respond to channel commands w/ predictable timing AMB's do not
also translates channel to DRAM commands but w/o additional scheduling check for DRAM protocol compliance does not protect against northbound frame collision
apparent strategy
minimize additional latency hit in the AMB daisy chain
already problematic due to the resync issue
maintain centralized control over scheduling and DRAM timing
AMB is less specialized for the DDRx DRAM component flavor AMB predictable timing response is required for this to happen anyway
Result
improve capacity & bandwidth, sacrifice latency
School of Computing University of Utah
19
CS7810
Sample Read & Write Transactions
A: RAS B:CAS and precharge DRAM RAS and posted CAS scheduled to different DRAM clocks Latency critical commands should be posted in slot A
Write data does not need to be contiguous allows read returns to be interleaved in a write burst, write command can precede completion of write data delivery
School of Computing University of Utah
20
CS7810
Page 10
AMB Asic
3 logic blocks
northbound pass-through southbound pass-through core
current write buffer design
buffer 32 72-bit write data frames
allows priority for read returns
all commands must be partially decoded
plus buffer the 3 write data frames that must be speculatively stored
since only 1-bit of the target AMB address is contained in each frame
CRC check & generate logic PISO (parallel in serial out)
serializes read returns into proper frame format on northbound lane
read return data is sync'd for seamless entry onto northbound lanes
removes rank switching overhead seen in conventional DDRx maximizes read bandwidth
21
School of Computing University of Utah
CS7810
Typical AMB Block Diagram
SMBus: Mem_Ctlr R/W access to configuration registers. Independent of high-speed N & S lanes. DOES NOT allow data access if northbound lanes fail
School of Computing University of Utah
22
CS7810
Page 11
Additional Features
BIST
for large capacity sequential testing is prohibitive BIST feature allows parallel test what is it really?
several autonomous FSM's configured via the SMBus
Thermal sensor
2-rank FB-Dimm and AMB consumes up to 20 watts
hence thermals can change rapidly need
protect the devices keep the thermal sensitive electrical properties in "open eye" status
FB mem_ctlr periodically reads the thermal sensor
throttles commands as necessary more centralized control
School of Computing University of Utah
23
CS7810
RAS Features
Reliability, Availability, Serviceability Checksum in the transport layer
CRC
particularly needed due to timing uncertainty
correct when a single bit lane loses phase resulting in burst loss on a single lane
Bit lane steering
lane failure happens
most commonly caused by DIMM socket interconnect failure
users put DIMMs in sockets uneven or ham-fisted pressure causes metal fatigue repeated thermal variations subsequently cause permanent failure
cure
for single lane failure steer remaining 9 lanes to the working lanes
School of Computing University of Utah
24
CS7810
Page 12
Steering Example
South lane failure example
alert frame sent north enters error wait FB ctlr sends soft reset
hence must keep copies of commands and data in flight run training sequence to discover faulty lane reconfigure registers via SMBus
failed lane does reduce CRC protection note
top and bottom lanes are not protected
School of Computing University of Utah
25
CS7810
Southbound Fail Over Mode
Command and write data example
normal 10 lanes & 120 bit frames
2 bits: frame type 24 bits: command 8 check or mask bits 22 bits of CRC 64 bits of dat...