How to compile MADCAP on a Linux PC cluster

Before you read this page, see my main MADCAP page for prerequisites.

I have tested this setup on an 11-node Linux PC cluster. The individual nodes have Athlon processors and have the standard Red Hat Linux 7.1 distribution installed. They have:

You can check that Red Hat LAPACK and BLAS packages are installed with the commands: 
> rpm -q lapack
> rpm -q blas
Rather than using the unoptimised libraries in the Red Hat packages, I recommend downloading ATLAS and installing the optimised LAPACK and BLAS libraries that it compiles.

LAM/MPI adds some wrapper programs mpicc and mpif77 that are used to call gcc and g77 with the correct options to compile and link with the MPI libraries.

Compiling MPIBLACS

Unpack mpiblacs.tgz and the patch mpiblacs-patch03.tgz, then move into the BLACS directory: 

> tar -zxvf mpiblacs.tgz
> tar -zxvf mpiblacs-patch03.tgz
> cd BLACS

Copy the Linux include file to the current directory: 

> cp BMAKES/Bmake.MPI-LINUX Bmake.inc

Edit Bmake.inc to reflect your setup. Some of the lines must be changed as follows: 

MPIdir = /usr/local

INTFACE = -Df77IsF2C

TRANSCOMM = -DUseMpi2

F77            = mpi77
CC             = mpicc

Make the libraries: 

> make mpi

Copy the libraries to so that they are named something a bit more useful: 

> cd LIB
> cp blacs_MPI-LINUX-0.a libblacsmpi.a
> cp blacsCinit_MPI-LINUX-0.a libcblacsmpi.a
> cp blacsF77init_MPI-LINUX-0.a libfblacsmpi.a

Move the libraries to a suitable library directory, hereinafter referred to as LIBDIR (for example, use a subdirectory of your home directory ~/lib): 

> mkdir LIBDIR
> mv *.a LIBDIR
> cd ../..

Compiling SCALAPACK

Unpack scalapack.tgz and move into the SCALAPACK directory: 

> tar -zxvf scalapack.tgz
> cd SCALAPACK

Copy the Linux include file to the current directory 

> cp INSTALL/SLmake.LINUX SLmake.inc

Edit SLmake.inc to reflect your setup. It should not require any changes, but make sure that the CDEFS entry is: 

CDEFS         = -Df77IsF2C -DNO_IEEE $(USEMPI)

Make the library: 

> make

Copy the library to LIBDIR: 

> cp libscalapack.a LIBDIR
> cd ..

Compiling MADCAP

Make a directory for madcap and unpack the madcap2.0.tar.gz in it: 

> mkdir madcap
> cd madcap
> tar -zxvf ../madcap2.0.tar.gz

In madcap.h, the correct definitions for this Linux platform need to be added. Replace the line 

#else
with 
#elif defined(LINUX)
  #define int4 int
  #define flt4 float
  #define flt8 double
  #define mad_char(s) s
  #define blacs_get blacs_get__
  #define blacs_gridinit blacs_gridinit__
  #define blacs_gridmap blacs_gridmap__
  #define blacs_gridinfo blacs_gridinfo__
  #define descinit descinit_
  #define pdlatra pdlatra_
  #define pddot pddot_
  #define pdgemv pdgemv_
  #define pdsymv pdsymv_
  #define pdtran pdtran_
  #define pdposv pdposv_
  #define pdposvx pdposvx_
  #define pdsyev pdsyev_
  #define pdsyevx pdsyevx_
  #define pdpotri pdpotri_
  #define pdpotrs pdpotrs_
  #define pdpotrf pdpotrf_
  #define dposv dposv_
  #define dpotri dpotri_
#else

The .c files in the QUAD directory also need a bit of editing. In each, change the line 

#include "madcap.h"
to 
#include "../madcap.h"

Now you can start compiling the code. Firstly compile madtool.c: 

> mpicc -DLINUX -c madtool.c

Next, change to both of the sub-directories MAP and QUAD in turn, and compile each of the .c files as follows. The MADCAP scripts expect the names of executables to end in .x. For example, qq.c: 

> mpicc -DLINUX -c qq.c 
> mpif77 -o qq.x qq.o ../madtool.o -LLIBDIR -lscalapack -lblacsmpi \
    -lcblacsmpi -lblacsmpi -llapack -lblas
or if you are using the ATLAS-compiled libraries: 
> mpicc -DLINUX -c qq.c 
> mpif77 -o qq.x qq.o ../madtool.o -LLIBDIR -lscalapack -lblacsmpi \
    -lcblacsmpi -lblacsmpi -llapack -lf77blas -lcblas -latlas
Note that -lblacsmpi appears twice (for some mysterious reason this is only needed by some of the programs).

MADCAP Scripts

The scripts that are used to run MADCAP are: 
MAP/map_bat
MAP/map_bug
MAP/map_script
QUAD/quad_bat
QUAD/quad_bug
QUAD/quad_script
Note that the line 
#!/bin/csh -f
must be on the top line of each of the files. That is not so in the scripts that come with the example data from the MADCAP site.

The scripts map_bug, map_bat, quad_bug and quad_bat need some changes. Change the variables for the directories so the scripts know the location of the MADCAP programs and your data. Change the line 

setenv MADRUN "mpprun -n $NO_PE"
to 
setenv MADRUN "mpirun -np $NO_PE"
Remember also to change the line 
setenv NO_PE 25
to reflect the number of processors you are going to use!

In Red Hat 7.1, csh is an alias to tcsh, which chokes on some of the scripts. Because of this, map_script and quad_script need some modifications.

In map_script, change 

  @ NO_Q = `wc -c inv_qq_noise.q_data`[1] / 4
to 
  @ NO_Q = `wc -c inv_qq_noise.q_data | awk '{ print $1 }'` / 4

In quad_script, just under 
# -----------------------------------
# Set Data Size Environment Variables
# -----------------------------------
change 
@ NO_PIX = `wc -c p_data`[1] / 4
to 
@ NO_PIX = `wc -c p_data | awk '{ print $1 }'` / 4
and 
set NO_BIN = `wc -w bin`[1]
to 
set NO_BIN = `wc -w bin | awk '{ print $1 }'`

[ Notes | Home ]

Last modified 2002/09/27

Mark Ashdown - maja1@mrao.cam.ac.uk