- •IBM Research
- •IBM Research
- •IBM Research
- •Blue Gene/L
- •IBM Research
- •IBM Research
- •IBM Research
- •IBM Research
- •IBM Research Simulation
- •IBM Research
- •IBM Research
- •IBM Research
- •IBM Research
- •IBM Research
- •IBM Research
- •IBM Research
- •IBM Research
- •IBM Research
- •IBM Research
- •IBM Research
- •IBM Research
- •IBM Research
- •IBM Research
- •IBM Research
- •IBM Research
- •IBM Research
- •IBM Research
- •IBM Research
- •IBM Research
- •IBM Research
- •IBM Research
- •IBM Research
- •IBM Research
- •IBM Research
- •IBM Research
- •IBM Research
- •IBM Research
- •IBM Research
- •IBM Research
- •IBM Research
- •IBM Research
- •IBM Research
- •IBM Research
- •IBM Research
- •IBM Research
- •IBM Research
- •IBM Research
- •IBM Research
IBM Research
Achieving Strong Scaling On Blue Gene/L: Case Study with NAMD
Sameer Kumar,
Blue Gene Software Group,
IBM T J Watson Research Center, Yorktown Heights, NY sameerk@us.ibm.com
© 2005 IBM Corporation
IBM Research
Outline
Motivation
NAMD and Charm++
BGL Techniques
–Problem mapping
–Overlap of communication with computation
–Grain size
–Load-balancing
–Communication optimizations
Summary
2 |
© 2005 IBM Corporation |
|
IBM Research
Blue Gene/L
© 2005 IBM Corporation
Blue Gene/L
System
64 Racks, 64x32x32
Rack
32 Node Cards
|
Node Card |
180/360 TF/s |
|
(32 chips 4x4x2) |
32 TB |
|
16 compute, 0-2 IO cards |
|
|
|
2.8/5.6 TF/s |
Compute |
Card |
512 GB |
|
||
2 chips, 1x2x1 |
|
|
Chip |
90/180 GF/s |
|
16 GB |
2 processors
5.6/11.2 GF/s
1.0 GB
2.8/5.6 GF/s
4 MB
IBM Research
Application Scaling
Weak
–Problem size increases with processors
Strong
–Constant problem size
–Linear to sub-linear decrease in computation time with processors
–Cache performance
–Communication overhead
• Communication to computation ratio
5 |
© 2005 IBM Corporation |
|
IBM Research
Scaling on Blue Gene/L
Several applications have demonstrated weak scaling
Strong scaling on a large number of benchmarks still needs to be achieved
6 |
© 2005 IBM Corporation |
|
IBM Research
NAMD and Charm++
© 2005 IBM Corporation
IBM Research
NAMD: A Production MD program
NAMD
Fully featured programNIH-funded development
Distributed free of charge
(thousands downloads so far)Binaries and source code
Installed at NSF centersUser training and support
Large published simulations
(e.g., aquaporin simulation featured in keynote)
8 |
© 2005 IBM Corporation |
|
IBM Research Simulation
NAMD, CHARMM27, PME NpT ensemble at 310 or 298 K
1ns equilibration, 4ns production
Protein: ~ 15,000 atoms Lipids (POPE): ~ 40,000 atoms
Water: ~ 51,000 atoms
Total: ~ 106,000 atoms
3.5 days / ns - 128 O2000 CPUs
11 days / ns - 32 Linux CPUs
.35 days/ns–512 LeMieux CPUs
F. Zhu, E.T., K. Schulten, FEBS Lett. 504, 212 (2001)
M. Jensen, E.T., K. Schulten, Structure 9, 1083 (2001)
9
IBM Research
Molecular Dynamics in NAMD
Collection of [charged] atoms, with bonds
–Newtonian mechanics
–Thousands of atoms (10,000 - 500,000)
At each time-step
–Calculate forces on each atom
•Bonds:
•Non-bonded: electrostatic and van der Waal’s
–Short-distance: every timestep
–Long-distance: using PME (3D FFT)
–Multiple Time Stepping : PME every 4 timesteps
–Calculate velocities and advance positions
Challenge: femtosecond time-step, millions needed!
10 |
© 2005 IBM Corporation |
|