Lecture CME212 13
Optimization Blockers, ILP, Pipelines
Function Side Effects
A function side effect occurs when you modify something other than the return value of the function
Global variables Function arguments (call-by-reference)
Example
d = max(dist(p,q),dist(q,r))
What if dist modifies q?
More on Side Effects
int func1(x) { return f(x)+f(x)+f(x)+f(x); }
int func2(x) { return 4*f(x); }
Is it safe to replace func1 by func2 to save 3 function calls?
Side Effects, contd
What if f() is defined as follows?
int counter = 0; int f(int x) { return counter++; } func1() will return 6 and func2() will return 0
const-correctness
In ANSI C you add the keyword const to arguments that are read-only
The memory object will never be written to
Can also be used for variables to make them into constants Helps the compiler to do analysis v is pointing to a readonly object const double pi = 3.14159; double norm(const double *v, int length)
Side Effect Example
if ( (x < y) || foo(x) )
Where x and y are floats. foo() is function returning an int (boolean). If foo(x) does not change its input, then there are cases (where x>=y) when the function foo() does not have to be evaluated (short-circuit evaluation) Saves work if foo() is expensive If foo() does change its input, if statement might not behave reliably
Branches
if-statements always introduce overhead, since a conditional must be evaluated jumps in the executable code (originating from ifstatements or loops) make life hard for modern processors (they disturb the instruction pipelining) Sometimes it is possible to reduce the overhead by performing the tests or evaluating the conditional in another way
If possible, use the case-construct (switch statement in C). This reduces the risk of redundant tests In an if-elseif-hierarchy or a case-construct, one or a few cases are often more frequent than the others. Test for these cases first, to reduce the average number of tests
Branches, contd
The evaluation of the conditional is normally performed from left to right, and the evaluation is stopped as soon as a decision can be made. This should be exploited when constructing the conditional: BAD:
if(i<ExpensiveComputation() && i>0)...
BETTER:
if(i>0 && i<ExpensiveComputation())...
SAFEST:
if(i>0) { if(i<ExpensiveComputation())... }
Loop-independent conditionals
Placing the conditional inside the loop should always be avoided if it is loop-independent
for(i=0;i<N;i++) if(a==0) b[i]=2*i; should be if(a==0) for(i=0;i<N;i++) b[i]=2*i;
Vector ADT
length data 0 1 2 length1
Procedures
vec_ptr new_vec(int len)
Create vector of specified length
int get_vec_element(vec_ptr v, int index, int *dest)
Retrieve vector element, store at *dest Return 0 if out of bounds, 1 if successful
int *get_vec_start(vec_ptr v)
Return pointer to start of vector data
Similar to array implementations in Pascal, ML, Java
e.g., always do bounds checking
Combine Operation
void combine1(vec_ptr v, int *dest) { int i; *dest = 0; for (i = 0; i < vec_length(v); i++) { int val; get_vec_element(v, i, &val); *dest += val; } }
Procedure
Compute sum of all elements of integer vector Store result at destination location Vector data structure and operations defined via abstract data type
Pentium II/III Performance: Clock Cycles / Element
42.06 (Compiled -g) 31.25 (Compiled -O2)
Understanding Loop
Inefficiency
Procedure vec_length called every iteration Even though result always the same
void combine1-goto(vec_ptr v, int *dest) { int i = 0; int val; *dest = 0; if (i >= vec_length(v)) goto done; loop: 1 iteration get_vec_element(v, i, &val); *dest += val; i++; if (i < vec_length(v)) goto loop done: }
Move vec_length Call Out of Loop
Optimization
Move call to vec_length out of inner loop
Value does not change from one iteration to next Code motion void combine2(vec_ptr v, int *dest) { int i; int length = vec_length(v); *dest = 0; for (i = 0; i < length; i++) { int val; get_vec_element(v, i, &val); *dest += val; } }
CPE: 20.66 (Compiled O2)
vec_length requires only constant time, but significant overhead
Code Motion Example #2
Procedure to Convert String to Lower Case
void lower(char *s) { int i; for (i = 0; i < strlen(s); i++) if (s[i] >= 'A' && s[i] <= 'Z') s[i] -= ('A' - 'a'); }
Lower Case Conversion Performance
Time quadruples when double string length Quadratic performance
lower1 1000 100
CPU Seconds
10 1 0.1 0.01 0.001 0.0001
256 512 1024 2048 4096 8192 16384 32768 65536 131072 262144
String Length
Convert Loop To Goto Form
strlen executed every iteration strlen linear in length of string
Must scan string until finds '\0'
void lower(char *s) { int i = 0; if (i >= strlen(s)) goto done; loop: if (s[i] >= 'A' && s[i] <= 'Z') s[i] -= ('A' - 'a'); i++; if (i < strlen(s)) goto loop; done: }
Overall performance is quadratic
Improving Performance
Move call to strlen outside of loop Since result does not change from one iteration to another Form of loop invariant code motion
void lower(char *s) { int i; int len = strlen(s); for (i = 0; i < len; i++) if (s[i] >= 'A' && s[i] <= 'Z') s[i] -= ('A' - 'a'); }
Lower Case Conversion Performance
Time doubles when double string length Linear performance
1000 100
CPU Seconds
10 1 0.1 0.01 0.001 0.0001 0.00001 0.000001
256 512 1024 2048 4096 8192 16384 32768 65536 131072 262144
String Length lower1 lower2
Optimization Blocker: Procedure Calls
Why couldnt the compiler move vec_len or strlen out of the inner loop?
Function may have side effects
Alters global state each time called
Function may not return same value for given arguments
Depends on other parts of global state Procedure lower could interact with strlen
Why doesnt compiler look at code for vec_len or strlen?
Linker may overload with different version
Unless declared static
Interprocedural optimization is not used extensively due to cost
Warning:
Compiler treats procedure call as a black box Weak optimizations in and around them
Reduction in Strength void combine3(vec_ptr v, int *dest)
{ int i; int length = vec_length(v); int *data = get_vec_start(v); *dest = 0; for (i = 0; i < length; i++) { *dest += data[i]; }
Optimization
Avoid procedure call to retrieve each vector element
Get pointer to start of array before loop Within loop just do pointer reference Not as clean in terms of data abstraction
CPE: 6.00 (Compiled -O2)
Procedure calls are expensive! Bounds checking is expensive
Eliminate Unneeded Memory Refs
void combine4(vec_ptr v, int *dest) { int i; int length = vec_length(v); int *data = get_vec_start(v); int sum = 0; for (i = 0; i < length; i++) sum += data[i]; *dest = sum; }
Optimization
Dont need to store in destination until end Local variable sum held in register Avoids memory 1 read, 1 memory write per cycle CPE: 2.00 (Compiled -O2)
Memory references are expensive!
General forms of combining
void abstract_combine4(vec_ptr v, data_t *dest) { int i; int length = vec_length(v); data_t *data = get_vec_start(v); data_t t = IDENT IDENT; for (i = 0; i < length; i++) t = t OP data[i]; *dest = t; }
Data Types
Use different declarations for data_t
int float double
Operations
Use different definitions of OP and IDENT + / 0 * / 1
Machine-independent Opt. Results
Method +
gcc -g gcc -O2 Move vec_length data access Accum. in temp 42.06 31.25 20.66 6.00 2.00
Integer *
41.86 33.25 21.25 9.00 4.00
Floating Point + *
41.44 31.25 21.15 8.00 3.00 160.00 143.00 135.00 117.00 5.00
When using multiplication, the combine operation corresponds to a factorial. FP units will overflow at i=34 and i=171. After overflow, you will only compute with NaNs which is much slower (compare to denorms). When accum. in temp an 80-bit register is used which does not overflow, see page 423-424 in OHallaron
Software Optimization
The compiler is our fundamental tool
Debuggers and profilers too!
Example of tasks
Remove optimization blockers Rule out pathological cases Remove unnecessary overheads
Inlining Algebraic optimizations Code motion
In some cases we must act as compilers
Requires deep understanding of both compilers and hardware
CPU Improvements
CPUs are made out of transistors
Technology
Moores law predicts that the number of transistors that can be put into a single chip doubles every 18th month. Incredibly small transistors
Currently we are at 90nm Will probably go to 65nm this year and maybe 45nm next year An atom is about 0.3nm
Small transistors have faster switching time
Speeding up CPUs
Faster transistors means higher performance More transistors means more logic to do intelligent things More transistors means more cache memory Increase clock frequency
The von Neumann Model
Memory Fetches instructions from the memory referenced by the program conter (PC) and computes results based on the data the instruction specified The Arithmetical-Logic Unit (ALU) does the actual work In this simple model, access times to memory is uniform
PC Registers
ALU
How long is a CPU cycle?
1982: 5MHz
200ns 60 m (in vacuum)
2002: 3GHz clock
0.3ns 0.3ns 10cm (in vacuum) 3mm (on silicon)
Instruction Stages
Instructions need to be fetched from memory
Encoded in the ISA of the chip
They have to be decoded
To know which parts of the CPU logic that are going to be used
Many instructions fetch memory into registers
Both reads and writes
Arithmetic work needs to be done once data is safely stored into the registers The different stages are performed using logic from different parts of the chip
Timing Signals traveling
Instruction Stages, Example
Instructions can often be divided into several independent parts (stages)
1. 2. 3. 4. 5. Instruction fetch (IF) Instruction decode/register fetch (ID) Execution/effective address (EX) Memory access (MEM) Write-back (WB)
Branch instructions require 2 stages, stores 4 stages and the rest 5
Observations
All steps are performed in succession
Steps are waiting for work most of the time Logic is idling
The average CPI is about 4.54 if we assume a branch frequency of 12% and a store frequency of 10% Solution: use pipelining
Start a new instruction each cycle Each of the clock cycles becomes a pipe stage Use all logic more
Real-World Pipelines: Car Washes
Sequential Parallel
Idea
Pipelined
Divide process into independent stages Move objects through stages in sequence At any given times, multiple objects being processed
Pipelines
1 Inst 1 IF Inst 2 Inst 3 Inst 4 Inst 5 2
ID IF
3
EX ID IF
4
MEM EX ID IF
5
WB
6
7
8
9
MEM WB EX ID IF MEM WB EX ID MEM WB EX MEM WB
Pipeline Speedup
For a 5 stage pipeline we can execute one instruction per cycle if pipe is full
5 times speedup
In general a pipe of length n will give a speedup of n if we ignore startup cost
Asymptotic performance
Same technique used in fabrication industry
Cars, mobile phones, etc.
Problems with Pipelines
Works fine if we can start a new instruction every clock cycle
Instructions are independent
If the pipeline detects a hazard it stalls, causing pipeline bubble Branch hazards
Instructions are started every clock cycle A branch can make instructions halfway through the pipe unneccessary The pipe needs to be flushed
Other hazards include structural hazards (chip resources) and data hazards (dependencies)
Example: We are out of steering wheels, must wait
Unusual to get a CPI of 1.0
More on Pipelines
Some operations, such as floating point operations, take much longer time to decode and operate upon
More logic used
Functional unit Int ADD Int MUL FP MUL FP DIV Typical cost in cycles 1 2 4 29
Dealing with Data Hazards
Dependent instructions might stall if
Memory access takes a long time We are out of registers Use a lot of logic (FP DIV)
We can fill the bubble with other instructions that do not depend on the stalling instruction
Instruction scheduling Decreases CPI Sometimes not possible
Today: ~10-20 stages and 4-6 pipes I Issue logic I I I R R R R Regs Mem B M MW B M MW B M MW
Instruction Scheduling
Compilers know which resources are available and how long instructions take Compilers schedule the instructions
Tries to minimize pipeline bubbles Keep all resources busy (increase parallelism)
Sometimes we will have bubbles due to hazards Can be improved using
Branch prediction Out-of-order execution
Branch Prediction
We can guess the outcome of a branch and let the branch instruction travel through the pipeline
Less bubbles
If our guess was wrong, we have to kill operations in the pipeline that depend on the misspredicted branch
Roll back
Very good accuracy
90-95% correct predictions for SPECcpu
Out-of-order Execution
Dynamic instruction scheduling
Overcomes data hazards
Lets independent instructions pass blocking instructions in the pipeline Size of reorder buffer determines how far in the future we can see
Putting it all Together
6-8 pipelines
2 load/store, 2 int, 4 floating point Complex
Many pipeline stages allow high clock frequencies
Small amount of logic in each step
Large out-of-order buffers
Complex Problems with interupts and exceptions
Advanced branch prediction
Complex
Lots of logic, still hard to get instruction-level parallelism
CPU Capabilities of Pentium III
Multiple Instructions Can Execute in Parallel
1 load 1 store 2 integer (one may be branch) 1 FP Addition 1 FP Multiplication or Division
Some Instructions Take > 1 Cycle, but Can be Pipelined
Instruction Latency Load/Store 3 Integer Multiply 4 Integer Divide 36 Double/Single FP Multiply 5 Double/Single FP Add 3 Double/Single FP Divide 38 Cycles/Issue 1 1 36 2 1 38
Limit of ILP