Set Hot Detection and Hardware Optimization -- report for the CS/ECE 752 project
Lei Chen & Su Zhang (chenl, [e-mail])
1. Project Goal
This project is to optimize and enhance performance of a superscalar processor. It starts with a baseline superscalar processor, modeled with the SimpleScalar toolset; 32KB RAM and four SPEC benchmarks: go, perl, cc1 and compress. The 32KB RAM is used for L1 caches, branch predictors and BTBs. The goal of the project is (1) find out the best hardware configuration with in the 32KB memory budget and (2) Add features to enhance the hardware performance.
2. Analysis
Table 1. lists the the best configuration we found in the warmup project. L1 Instruction Cache L1 Data Cache L2 Cache Brach predictor 8192 entries, gshare BTB 512 entries, 32-way associativity Table.1 best baseline configuration. First, we will summarize some observations made in the warmup project : 1. Increasing the cache associativity helps to reduce the conflict cache miss and hence improve the performance. However, there is little benefit when the associativity is greater than 8. 2. The instruction cache need to more cache than the data cache. It is because the data cache miss will only stall the execution of those instructions depending on it. However, instruction cache miss will stall the code fetch of all following instruction and potentially stalls the whole pipeline. The other reason is that instruction cache shows greater spatial locality than data cache. The LRU replace algorithm is more efficient for the Instruction 16K, 8-way associativity, 256B cacheline 8K, 8-way associativity, 64B cacheline 128K unified, 4-way associativity, 256KB cacheline 2048 entries, comb + 4096 entries, bimod predictor +
cache. 3. The precise branch prediction is critical to the performance. Increasing the size of prediction entries helps to solve the alias confliction. However, the precision of the branch predictor also depends one the nature of the predict algorithm, An infinite number of prediction entries still can not guarantee a prefect predictor. In order to find a good direction to do the optimization, we need to find out which component has more potential to increase the performance. A good way to do this is compare the performance of the baseline configuration with the Ideal configuration. Figure.1 shows the IPCs in different configurations. From here we can make the following observations: 1. The hybrid predictor of bimetal and gshare does well in the branch prediction. However, compared with a perfect predictor, there are still chances to improve the performance by increase the precision of the predictor. In the Figure. 1, The configuration with prefect predictor outperforms other configurations by a great deal. 2. 8-way associative cache has almost the same performance as a fully associative cache. So there is little incentive to optimize the cache for our baseline configuration. However, as the speed of CPU continues to increase in the speed much faster than the memory. The most essential characteristic for a L1 cache its access speed. For example, in Pentium 4 processor[9], the L1 cache must provide a word in only 2 cycles. In An 8-way associative cache, there are complicate circuits dealing with the tag comparison. So the cache access speed is limited. One solution for this problem is using low associative cache or even using direct-mapped cache. Direct mapped cache suffer with thrashing when two different memory locations are accessed alternatively. Figure.1 shows that the IPC of Direct mapped L1 cache configuration is much lower than the IPC of 8-way associative cache(Although in the reality, the access latency of 8-way associative cache can not be just one cycle!). Figure. 2 shows the comparison of L1 cache miss rate. Instead of doing some optimization over a too-fancy configuration, we prefer to do some realistic work. It will be more meaningful optimize a direct mapped cache and we believe the result will reveal some interesting conclusion.
Figure.1 The IPCs for different configurations. The Idea_both use perfect predictor and fully associative cache (16K L1 i-cache, 8K L1 d-cache, 128K L2 unfied cache) . The Ideal_pred only uses perfect predictor and the ideal_asso only uses fully associative cache. The base is the configuration listed in Table.1 and The direct_map uses direct-mapped L1 cache.
Figure.2 The comprison of L1 cache miss rate between direct mapped cache and 8-way associative cache
3. Design Philosophy
With the limited RAM resources, we think the right way to maximize the performance is to dynamically change the allocation of resources to adapt to the program behavior and put the resources on key problems. With this goal in mind, we divide the problem into 3 parts. The first one is to explore how to dynamically change the configuration of cache to reduce the miss rate. The program execution can be seen as moving through a series of working phrases, and in each phrase, the misses are not uniformly distributed among caches sets. So we are inspired by the idea of dynamic analysis of program working sets, and we generalize it to detect the miss rate
pattern in each work phrase and try to find ways to dynamically relocate cache resources to hot sets. The second part is to see what is the key problem in the branch predictor and how to optimize the predictor by fighting those key problem within limited resources. So basically we realize the negative aliasing may be a big factor in predictor and we implement the YAGS to see how well it performs comparing other schema. The third part is to explore how to boost the performance with technique requiring little RAM resources like prefetching. For this project, it turns out we only have time to finish the first two parts, and we consider the third part as our future work.
4. Victim Buffer
Victim buffer is a fully-associated cache usually between L1 cache and L2 cache. It is aimed to improve the performance of direct mapped cache by avoiding the thrashing. With victim buffer, no data line appears both int the direct-mapped cache or victim buffer. In the case of a cache miss, the victim buffer is checked. If the missed line is in the victim buffer, The contents of the direct-mapped cache line and the matching victim buffer line are swapped. If the missed line is also not in the victim buffer, the data is loaded for the lower level cache. And the victim line of the direct mapped cache will evict one line in the victim buffer. Figure.3 shows the organization of the victim buffer. The experiment result shows that the victim buffer does improve the cache performance. The miss rate of victim buffer is between the miss rate of direct mapped cache and 2 way associative cache. In some case, the miss rate is very near the 2 way associative cache. This is due to two reasons. 1. The victim buffer actually increase the size of the cache. In some senses, it will remove some capacity misses because now the data that can be hold is the capacity of the direct mapped cache plus the size of the victim buffer. However, since the victim buffer is usually much small than the size of the direct mapped cache, this should be the second order factor.
From Processor
To Processor
Tag address to lower cache MRU entry tag tag tag LRU entry tag comparator comparator comparator comparator
Data
Direct Mapped Cache
data data data data
data from lower cache Fully-associativity victim buffer
Figure.3 Victim Buffer Organization (From paper [6])
2. The victim buffer reduces the thrashing. Suppose we have two instructions I1 and I2. They happen to be mapped into the same line in the direct mapped cache. If the program is executing in the pattern of I1, I2, I1, I2...., with solely direct mapped cache, each instruction fetch will cause a cache miss. With the victim buffer, I1 will be put to the victim buffer when the first time I2 is referenced. And after that, Each reference will only cause a victim buffer swap, which is much faster than to access the lower level cache. The draw back of victim buffer is that it does not scale well as the size of the direct mapped cache increases. Since the victim buffer is fully associated, enlarging the size of victim buffer will also prolong the latency to access the victim buffer and hence increase the miss penalty. The Figure.4 shows a statistics about the access number and misses number of the L1 data cache for Go benchmark. The X axis denotes the cache lines and the Y axis denotes the access number and miss number. For the figure we can see that the variation on both access numbers and miss numbers are great.
Fig.4 The miss rate of L1 data cache. The x axis denote the different lines in the direct mapped cache. The left figure shows the miss # for different lines. The right figure shows the access # for different lines.It is obvious that there exists some hot set int the cache, where both the access # and misses # is much higher than the average.
In the next part of this section, we propose two approaches to solve the drawback of victim buffer. Both approaches are based on multistage auto-reconfiguration. Under this scheme, The hardware will do some simple and fast statistics operations in the run time. Over certain time interval, the system gather this statistics, identify the cache hot sets and then reconfigure the cache. In the remaining of this section, we will describe these two approaches in detail.
4.1 Proportional Sharing Victim Buffer
After we study the working sets on miss rate of cache sets, we can notice there are always some sets have much greater miss rate than other sets. So we think that if we give more fraction of entries to those hottest and keep the cold sets kicking out entries used by hottest, we may reduce the total miss rate. And the whole execution can be identified into several working sets in which the miss rate pattern can keep rather stable, so we will make reconfiguration of victim butter at the end of working sets. For the hardware implementation simplicity, we use the proportional sharing of victim buffer to constrain the possible replacement of hottest by cold sets. The way we achieve the above policy is following. We rank the sets according to the miss rate for each set, and when an entry will replace some other entry in the victim buffer, we will constrain their access with a possibility from their ranking. So from a long run view, victim buffer will hold more entries from hot sets, which reduce the miss rate of them.
We allow it to replace entries in Victim buffer with possibility genereated by its ranking
Cache block 0x5555
Rank 10 Victim buffer
Figure.5 Illustration of the proportional sharing victim buffer control
But the results turn out that we have not gained anything by doing this, some time we even increase the total miss rate. Then we did more analysis to the behavior of victim buffer to figure out the reason behind it. Actually victim buffer just functions as more associatively to some sets of cache. But if two conflicting accesses to the same entry may not be hold by victim buffer if there are much more other accesses to the victim buffer happened between them. So only after we identify how many confliction there are in execution and how many of them could possibly be saved by victim buffer, we will know more clearly about the reason behind our experiment. First we plotted the transient conflicts and total conflicts for every set of cache. First we separate transient conflicts from total potential conflicts. The transient confliction means there are no other accesses between them (to this entry) and the evictions in the victim buffer happened between them will not exceed the victim buffer size which means the victim buffer can potentially save this confliction. Contrast to transient conflicts, long distant conflicts means there are several other accesses between these two identical accesses. The total conflicts within victim buffer means optimal number of conflicts victim buffer can save, which include all conflicts, even those same accesses to one entry are separated by several other accesses, so long the number of these different accesses is within the space of victim buffer, we also count them in.
Figure.6 The behavior of the L1 I-cache for benchmark Go.
Figure.7 The behavior of L1 D-Cache for Go benchmark.
From the figure 6 and 7, we can clearly see the differences of conflicts between each entry in the cache, and most of them are transient conflicts(the figure for other benchmark are similar). Actually we also plot the conflicts for every distance between two identical access, we find them are much less than transient, so we just group them together. Then we plot the normal victim buffer hits counts comparing to them. The VB hits show how many conflicts normal victim buffer saved.
Figure .8 Transient Conflicts VS Buffer hits VS Potential Buffer benefits
From the Figure.8 we can see, without any modification, the normal victim buffer already save most of conflicts and even sometime they can save all the transient conflicts and some other multiple distance conflicts. And because the conflicts with long distance, which are much more less than transient ones, need more entries from victim buffer, and since the entries in victim buffer are far less than the total entries in cache, keeping them will kick out transient conflicts from cold set. Although for one cold set, the transient conflicts are not many, but the total transient conflicts can not be ignored. So we will suffer a lot from keeping long distant conflicts of hotsets. So what we can do is just to try our best to save transient conflicts from any sets, which is already well done by normal victim buffer. So we conclude that we may not achieve more than normal victim buffers, and dynamically adapting other resources usage to program behavior may bring us more gains than victim buffers.
4.2 Direct Mapped Victim Buffer
From Processor tag tag tag tag tag tag tag tag set bit set bit to next lower cache To Processor data data data data data data data data tag tag cacheline cacheline
First level Direct map cache
data from lower cache Second level Direct mapped victim buffer
Figure.9 Direct mapped Victim Buffer Organization
One drawback of victim buffer is that it is fully-associated. The access latency for the victim buffer is high and the comparison circuits are complicated. So a improvement is to use direct mapped victim buffer instead. Figure. 9 show the organization of the direct mapped victim buffer. The most difference between direct mapped victim buffer and normal victim buffer is that for each cache line, there is only one victim buffer associated with it. When there is a cache miss, the replaced line always go to the same line in the victim buffer. Although, this approach accelerate the victim buffer access speed. There are two main problems: 1. Direct mapped victim buffer does not help when there are more than two data competing for the same cache line. 2. The miss coming for different cache lines may interfere with each other when they compete for the same victim buffer line. For the first problem, we have no found a effective way to solve it. However, this case does not happen very frequently in our benchmarks. We solve the second problem by detecting the hot set of the cache. Each cache line is associated with a counter. When miss happens in a cache line, the counter associated with that cache line increases by one. On every 1 million instructions, we check the the counters and can find out which line is hot and which line is
cold. We only enable hot lines to access the victim buffer. The reason of doing so is that the probability for a cold line to find back its data in the victim buffer is low. So there is no reason for the data from this kind of lines to evict other hot lines in the victim buffer. The Figure.10 shows the miss rates for direct mapped enhanced by the fully-associative victim buffer, direct mapped victim buffer and the miss rate of 2-way associative cache. We can see that the direct mapped victim buffer is almost as good as the fully-associative victim buffer.
Figure.10 The miss rates for the victim buffer, direct victim buffer and 2-way associative cache.
6. Exploring Branch Predictor
In this section, We explore how we could increase the prediction rate within limited resources. As we have stated, there are many factors affects the prediction rate, and from the survey of papers we think the negative aliasing is most important to two level branch predictors. It is because the limited entries in the predictor cause several addresses mapping to the same place and negatively affect each others direction prediction. In order to reduce the negative aliasing, either we could increase the PHT size or we could use some resources to keep information about those confliction addresses and handle them especially. So we adopt the idea of YAGS which just fit our need. YAGS has a bimodal choice PHT and two tag tables which keep the potential aliasing addresss tags. When we approach the choice PHT, we also look up the contradict cache table to see whether we have some matched tag, if so we will use the predictor provided by them. We also replace the gShare part of hybrid predictor to see the difference. And to our surprise, YAGS does not outperform the hybrid (bimodel and gShare) predictor in our experiment as showing in Figure 11.
Figure. 11 Direction Prediction Rates for CombYags, Yaks and Comb2lev
Actually the hybrid predictor is aiming the problem of slow adaptation of Global history predictors to the context switch. But YAGS has a bimodel predictor as its major direction table; it actually already functions the same way as the bimodal in the hybrid predictor. So putting YAGS into hybrid will achieve little gains. From figure 2, we can compare the fraction of usage for bimodal predictor in hybrid, and we can see that hybrid
predictor with gShare benefit a lot from bimodal contradicting to the hybrid with YAGS.
Figure.12 Choice distribution for different predictors
We also found that the experimental result of YAGS paper (the figure 9 and figure 13 in [8]) does not comply with our experiment (show in Figure 13). GShare shows better prediction rate and YAGS predictor only increase Gos prediction rate. It may be caused by the recursions in the Go whose instructions in different recursion levels conflict with each other.
Figure. 13 gShare VS YAGS for 2K size
7.Result Summary
Table.2 shows the final configuration of our system. L1 I-Cache L1 D-Cache Baseline Configuration Brach predictor BTB Enhanced Features Victim buffer L2 Cache 16K, Direct mapped, 256B cacheline 8K, Direct mapped, 64B cacheline 128K unified, 4-way associativity, 256KB cacheline 2048 entries, ...