Lecture 04: Embarrassingly Parallel, or Not

COSC 272: Parallel and Distributed Computing

Spring 2023

Announcements

1. Programming Assignment 01 Posted
   • ignore HPC cluster part of assignment for Friday
   • accounts registered, but no documentation yet
   • visit hpc.amherst.edu
   • ssh access: [amherstid]@hpc.amherst.edu
2. First written assignment next Friday
   • posted this weekend
3. Office Hours
   • TA (Mary Kate) Office Hours Wednesday 7–9pm, SCCE C109
   • My individual OH: Thursday 1:00–2:30

Outline

1. Lecture 03 Activity
2. Parallelism vs Concurrency
3. Embarrassingly Parallel Problem
4. Limitations of Parallelism

Lecture 03 Activity

void increment(int[] a) {
    int i = 0;
    while (i < a.length) {
        a[i] = a[i] + 1;
        i = i + 1;
    }
}

Question 1

If a = [0, 0, 0, 0] and two threads, what are possible outcomes?

void increment(int[] a) {
    int i = 0;
    while (i < a.length) {
        a[i] = a[i] + 1;
        i = i + 1;
    }
}

Question 2

If a = [0, 0, 0, 0] and $k$ threads, what are possible outcomes?

void increment(int[] a) {
    int i = 0;
    while (i < a.length) {
        a[i] = a[i] + 1;
        i = i + 1;
    }
}

Parallelism vs Concurrency

Concurrency performing multiple tasks that occupy overlapping time intervals

• E.g., I teach COSC 225 and COSC 273 concurrently

Virtues and Perils

Parallelism can give performance boost

• performance is one focus of this class

Back to Counter

public void increment () {
    ++count;
}

How could we fix the problem of mis-counting?

• Want every increment to count!

“Easy” Solution

Each thread stores own private count!

• run threads until they’re done
• aggregate local counts when threads terminate

Question

When might “easy” solution not be sufficient?

Embarrassingly Parallel Problems

A computational problem is embarrassingly parallel if it can be broken into many simple computations, (almost) all of which can be performed in parallel.

Example: Monte Carlo Estimation

A Formula from High School

Area of a disk: $A = \pi r^2$

An Idea from Probability

Pick a random point inside the framed region.

The probability the point lies in the disk is proportional to the disk’s area.

In More Detail

• area of disk is $\pi r^2$
• area of surrounding square is $(2 r)^2 = 4 r^2$
• the probability that a (uniformly) random point in the square lies in the disk is: $ \frac{\text{area of circle}}{\text{area of square}} = \frac{\pi r^2}{4 r^2} = \frac 1 4 \pi. $

so…

Estimation by Sampling

…to estimate $\pi $, suffices to estimate the probability that a random point point in the square lies inside the disk:

• pick a bunch of random points
• see how many lie in disk
• $p = $ proportion of points that do
• $\pi \approx 4 p$

Example of Monte Carlo method

Question

Why is Monte Carlo estimation embarrassingly parallel?

Another Question

How much performance increase with $k$ cores?

Not So Parallel

Dependencies?

a1 = b1 + c1;
a2 - b2 + c2;
d = a1 * a2

More Generally

Consider a program that requires

• $N$ elementary operations
• $T$ time to run sequentially

Suppose

• a $p$-fraction of operations can be performed in parallel
• $1-p$ fraction must be performed sequentially

Question: how long could program take with $n$ parallel machines?

Idea

With $n $ parallel machines:

• perform $p $-fraction of parallelizable ops in parallel on all $n$ machines
  • total time $\frac{T \cdot p}{n}$
• perform remaining ops sequentially on a single machine
  • total time $T \cdot (1 - p)$

Total time: $T \cdot (1 - p) + T \cdot \frac{p}{n} = T \cdot \left(1 - p + \frac p n\right)$

How Much Improvement?

The speedup is the ratio of the original time $T $ to the parallel time $T \cdot \left(1 - p + \frac p n\right)$:

• $S = \frac{1}{1 - p + \frac p n}$

This relation is called Amdahl’s Law

Example

1 person can chop 1 onion per minute

Recipe calls for:

• chop 6 onions
• saute onions for 4 minutes

Note:

• chopping onions can be done in parallel
• sauteing
  • takes 4 minutes no matter what
  • must be accomplished after chopping

Example (continued)

How much can the cooking process be sped up by $n $ cooks?

Example (continued)

• For one chef, $T = 6 + 4 = 10$
• Only chopping onions is parallelizable, so $p = 6 / 10 = 0.6$
• Amdahl’s Law:
  • $S = \frac{1}{1 - p - \frac{p}{n}} = \frac{1}{0.4 + \frac 1 n 0.6}$
• So:
  • $n = 2 \implies S = 1.43$
  • $n = 3 \implies S = 1.67$
  • $n = 6 \implies S = 2$
• Always have $S < 1 / (1 - p) = 2.5$

Speedup Improvement by Adding More Processors

• Second processor: 43%
• Third processor: 17%
• Fourth processor: 9%
• Fifth processor: 6%
• Sixth processor 4%

Latency vs Number of Processors

How does latency $T $ scale with $n $?

• Adding more processors has declining marginal utility:
  • each additional processor has a smaller effect on total performance
  • at some point, adding more processors to a computation is wasteful
• Another consideration:
  • after parallel ops have been performed, extra processors are idle (potentially wasteful!)

Remarks

The proportion of parallelizable operations $p$ is not always obvious from problem statement

Next Time

Start Mutual Exclusion

• How can we fix our Counter to work as intended if we need to maintain a running count that can be accessed by multiple threads?