25 July 2012

216. Quantum Espresso on ROCKS 5.4.3 / CentOs 5.6 -- Almost got it.


I've seen just about any error message imaginable when compiling Quantum Espresso. You can most likely get around them by following this guide ad verbatim 
(no parallel environment detected, can't build fortran programs, no linker, parallel vs serial compiler etc.)

This compile is nothing like the easy-breezy one on debian, because of one reason: CentOS 5.6 is old. It's so old that the default gcc version can't compile QE. You'd encounter similar problems with an increasing number of software packages, including ABINIT.

      CentOS 5.6: gcc version 4.1.2 20080704 (Red Hat 4.1.2-50)
      Wheezy:       gcc version 4.7.1 (Debian 4.7.1-2)

The crusty compiler collection means we have to hammer out errors like these during compilation:

In file dfile_star.f90:28
    INTEGER,ALLOCATABLE ::  npert (:), irgq (:)
Error: Attribute at (1) is not allowed in a TYPE definition
 In file dfile_star.f90:37

    REAL(DP),ALLOCATABLE :: gi (:,:), gimq (:), eigen(:)
Error: Attribute at (1) is not allowed in a TYPE definition
 In file dfile_star.f90:42

    COMPLEX(DP),ALLOCATABLE :: u(:,:), t(:,:,:,:), tmq (:,:,:)
Error: Attribute at (1) is not allowed in a TYPE definition

Hence the requirement to first compile GCC 4.7, even if it takes a couple of hours. 

To the students out there swearing (and sweating) over their professors' insistence on using ROCKS/CentOS: compiling a compiler is not a wasted skill and just might be useful in building cross compilation environments for the time when you decide that science can take a hike and there's more money in working on setting up embedded systems. At least it'll buy you a little geek cred. 

Also, it seems that you need to use math and mpi libs compiled with the same compiler as you use to compile quantum espresso. So...lots of steps today as well!

Also, I haven't reported the configure bug for openmpi 1.6. Feel free to reproduce and report it.

I'm not happy that Quantum Espresso ignores the compile flags I've given it and defaults to things it should not default. I've had to jump through way too many hoops to be happy about this. OpenMPI not supporting program-suffix annoys me as well.

This is another massive compile...I'd rate it as my most annoying, troublesome and infuriating one yet.

1. Build GCC (gfortran, g++, gcc) 4.7 according to this post:
Make sure that
export PATH=$PATH:/share/apps/tools/gcc/gcc47/bin

2. Create target directory structure
sudo mkdir /share/apps/libs
sudo chown $USER /share/apps/lib

3. Recompile GNU OpenMP (libgomp)
It's probably easier you start with a freshly extracted gcc-4.7.1
cd ~/tmp/gcc
mkdir newgcc
cp gcc-4.7.1.tar.gz newgcc/
cd newgcc
tar xvf gcc-4.7.1
cd gcc-4.7.1/libgomp
mkdir build
cd build/
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/share/apps/tools/gcc/mpc/lib:/share/apps/tools/gcc/gmp/lib:/share/apps/tools/gcc/mpfr/lib
.././configure CC=gcc-gcc-4.7 CXX=g++-gcc-4.7 F77=gfortran-gcc-4.7 FC=gfortran-gcc-4.7 --prefix=/share/apps/tools/gcc/gcc47

md5sum /share/apps/tools/gcc/gcc47/lib/libgomp.so.1.0.0
23db68121aacf1e7d896704474e26350  /share/apps/tools/gcc/gcc47/lib/libgomp.so.1.0.0

make install

md5sum /share/apps/tools/gcc/gcc47/lib/libgomp.so.1.0.0
791a1ceb0ea4009e22703b82e8ce1ff4  /share/apps/tools/gcc/gcc47/lib/libgomp.so.1.0.0

4.  Compile OpenMPI using your new GCC 
cd ~/tmp/
wget http://www.open-mpi.org/software/ompi/v1.6/downloads/openmpi-1.6.tar.gz
tar xvf openmpi-1.6.tar.gz
cd openmpi-1.6
mkdir build
cd build/
export LD_LIBRARY_PATH=/share/apps/tools/gcc/gcc47/lib64:/share/apps/tools/gcc/gcc47/lib/gcc/x86_64-unknown-linux-gnu/4.7.1:/share/apps/tools/gcc/gmp/lib:/share/apps/tools/gcc/mpc/lib:/share/apps/tools/gcc/mpfr/lib

.././configure --prefix=/share/apps/libs/openmpi47 CC=gcc-gcc-4.7 FC=gfortran-gcc-4.7 F77=gfortran-gcc-4.7 F90=gfortran-gcc-4.7 CPP=cpp-gcc-4.7 CXX=g++-gcc-4.7 --with-sge LDFLAGS="-L/share/apps/tools/gcc/gcc47/lib64 -L/share/apps/tools/gcc/gcc47/lib/gcc/x86_64-unknown-linux-gnu/4.7.1 -L/share/apps/tools/gcc/gmp/lib -L/share/apps/tools/gcc/mpc/lib -L/share/apps/tools/gcc/mpfr/lib -L/share/apps/tools/gcc/gcc47/lib" CPPFLAGS="-I/share/apps/tools/gcc/mpc/include -I/share/apps/tools/gcc/gmp/include -I/share/apps/tools/gcc/mpfr/include -I/share/apps/tools/gcc/gcc47/lib/gcc/x86_64-unknown-linux-gnu/4.7.1/include"

** There's a bug in the openmpi configure script leading to config failure due to to an extra space being introduced in front of -L when testing the F77 compiler. Exporting the LD_LIBRARY_PATH is done to get around that. **

make install

cd /share/apps/libs/openmpi47bin/
ls mpi* |xargs -I {} ln -s {} {}-gcc-4.7
echo 'export PATH=$PATH:/share/apps/libs/openmpi47/bin' >> ~/.bashrc
source ~/.bashrc

cd /share/apps/libs/openmpi47share/openmpi/
ln -s mpif77-vt-wrapper-data.txt mpif77-gcc-4.7-vt-wrapper-data.txt
ln -s mpif90-vt-wrapper-data.txt mpif90-gcc-4.7-vt-wrapper-data.txt
ln -s mpicc-vt-wrapper-data.txt mpicc-gcc-4.7-vt-wrapper-data.txt
ln -s mpic++-vt-wrapper-data.txt mpic++-gcc-4.7vt--wrapper-data.txt

ln -s mpicxx-vt-wrapper-data.txt mpicxx-gcc-4.7-vt-wrapper-data.txt
ln -s mpif77-wrapper-data.txt mpif77-gcc-4.7-wrapper-data.txt
ln -s mpif90-wrapper-data.txt mpif90-gcc-4.7-wrapper-data.txt
ln -s mpicc-wrapper-data.txt mpicc-gcc-4.7-wrapper-data.txt
ln -s mpic++-wrapper-data.txt mpic++-gcc-4.7-wrapper-data.txt
ln -s mpicxx-wrapper-data.txt mpicxx-gcc-4.7-wrapper-data.txt

5. Compile FFT using your new GCC
cd ~/tmp
wget ftp://ftp.fftw.org/pub/fftw/fftw-3.3.2.tar.gz
tar xvf fftw-3.3.2.tar.gz

cd fftw-3.3.2/
./configure --enable-float --enable-mpi --enable-threads --with-pic --prefix=/share/apps/libs/fftw-3.3.2-gcc47/single CC=gcc-gcc-4.7 FC=gfortran-gcc-4.7 CPP=cpp-gcc-4.7 MPIF90=mpif90-gcc-4.7 F77=gfortran-gcc-4.7 MPICC=mpicc-gcc-4.7 MPIF77=mpif77-gcc-4.7

make install
make check

         FFTW threaded transforms passed basic tests!
Executing "mpirun -np 1 ..
make[3]: *** [check-local] Error 1

That mpirun won't work isn't that surprising since it's still point to /opt/openmpi/bin instead of our mpirun-gcc-4.7

make distclean
./configure --disable-float --enable-mpi --enable-threads --with-pic --prefix=/share/apps/libs/fftw-3.3.2-gcc47/double CC=gcc-gcc-4.7 FC=gfortran-gcc-4.7 CPP=cpp-gcc-4.7 MPIF90=mpif90-gcc-4.7 F77=gfortran-gcc-4.7 MPICC=mpicc-gcc-4.7 MPIF77=mpif77-gcc-4.7

make install
make check
         FFTW threaded transforms passed basic tests!
...followed by the same error as above.
It's fine.

6. Optional: Compile openblas using your new GCC
cd /share/apps/tools/gcc/binutils/bin
ln -s ar-gcc-4.7 gcc-ar

cd ~/tmp
wget http://nodeload.github.com/xianyi/OpenBLAS/tarball/v0.1.1
mv v0.1.1 openblas.tar.gz
tar xvf openblas.tar.gz
cd xianyi-OpenBLAS-e6e87a2/
wget http://www.netlib.org/lapack/lapack-3.4.1.tgz

export LD_LIBRARY_PATH=/share/apps/tools/gcc/gmp/lib:/share/apps/tools/gcc/mpc/lib:/share/apps/tools/gcc/mpfr/lib

make all BINARY=64 CC=gcc-gcc-4.7 FC=gfortran-gcc-4.7 MPICC=mpicc-gcc-4.7 USE_THREAD=0 INTERFACE64=1 

make PREFIX=/share/apps/libs/openblas47 install

cp libopenblas_* /share/apps/libs/openblas47/lib
cd /share/apps/libs/openblas47/lib
ln -s libopenblas_nehalem-r0.1.1.so libopenblas.so.1
ln -s libopenblas.so.1 libopenblas.so
ln -s libopenblas.so libopenblas.so.0
ln -s libopenblas_nehalem-r0.1.1.a libopenblas.a

7. Compile Quantum Espresso using your new GCC
sudo mkdir /share/apps/QE
sudo chown $USER /share/apps/QE
mkdir /share/apps/QE/lapack-3.2

mkdir ~/tmp/QE
cd ~/tmp/QE

wget http://qe-forge.org/frs/download.php/211/espresso-5.0.tar.gz
wget http://qe-forge.org/frs/download.php/214/PWgui-5.0.tgz
wget http://qe-forge.org/frs/download.php/204/xspectra-5.0.tar.gz

Edit environmental_variables
PARA_PREFIX="mpirun-gcc-4.7 -n 8"

Time to configure and compile:
cd ~/tmp/QE
tar xvf espresso-5.0.tar.gz
cd espresso-5.0/
cp lapack-3.2 -R /share/apps/QE/
cp BLAS -R /share/apps/QE/

Create a script called compile.sh:

make clean
export PATH=""
export PATH="/usr/bin:/usr/sbin:/bin:/sbin:/usr/local/ecce/apps/scripts/share/apps/pvm/pvm3/bin/LINUX64:/share/apps/libs/openmpi47/bin:/share/apps/nwchem/nwchem-6.1/bin/LINUX64:/share/apps/sinfo/bin:/share/apps/sinfo/sbin:/share/apps/gromacs/bin:/share/apps/cmake/bin:/share/apps/tools/babel/bin:/share/apps/tools/htop/bin:/share/apps/tools/xmgrace/grace/bin:/share/apps/tools/rasmol/src:/share/apps/tools/strace/bin:/share/apps/tools/gcc/gcc47/bin:/share/apps/tools/gcc/binutils/bin"
which mpif90
read -p "press any key"
export CC=gcc-gcc-4.7
export FC=gfortran-gcc-4.7
export MPIF90=mpif90-gcc-4.7
export MPIF77=mpif77-gcc-4.7
export F77=gfortran-gcc-4.7
export F90=gfortran-gcc-4.7
export CPP=cpp-gcc-4.7
export CXX=g++-gcc-4.7
export CFLAGS=""
export CPPFLAGS="-I/share/apps/tools/gcc/mpc/include -I/share/apps/tools/gcc/gmp/include -I/share/apps/tools/gcc/mpfr/include -I/share/apps/tools/gcc/gcc47/lib/gcc/x86_64-unknown-linux-gnu/4.7.1/include -I/share/apps/libs/openmpi47/include -I//share/apps/tools/gcc/gcc47/lib/gcc/x86_64-unknown-linux-gnu/4.7.1/include"
export LDFLAGS="-L/share/apps/tools/gcc/gcc47/lib -L/share/apps/tools/gcc/gcc47/lib64 -L/share/apps/tools/gcc/gcc47/lib/gcc/x86_64-unknown-linux-gnu/4.7.1 -L/share/apps/tools/gcc/gmp/lib -L/share/apps/tools/gcc/mpc/lib -L/share/apps/tools/gcc/mpfr/lib -L/share/apps/libs/openmpi47/lib -L/lib64 -lc"
#export BLAS_LIBS="-L/share/apps/libs/openblas47/lib -lopenblas" 
export BLAS_LIBS="/share/apps/QE/BLAS/blas.a"
export LAPACK_LIBS="/share/apps/QE/lapack-3.2/lapack.a"
export FFT_LIBS="-L/share/apps/libs/fftw-3.3.2-gcc47/double/lib -lfftw3 -lfftw3_mpi -lfftw3_threads"
export MPI_LIBS="-L/share/apps/libs/openmpi47/lib -l:/share/apps/libs/openmpi47/lib/libmpi.so"
export LD_LIBRARY_PATH=/share/apps/tools/gcc/gmp/lib:/share/apps/tools/gcc/mpfr/lib:/share/apps/tools/gcc/mpc/lib:/share/apps/libs/openmpi47/lib
./configure --prefix=/share/apps/QE/bin|tee conf.log
read -p "Press key to continue"
cd PW
make |tee make.log
cd ../
make all |tee makeall.log

Then run it using

sh compile.sh

Which gives:

ESPRESSO can take advantage of several optimized numerical libraries
(essl, fftw, mkl...).  This configure script attempts to find them,
but may fail if they have been installed in non-standard locations.
If a required library is not found, the local copy will be compiled.

The following libraries have been found:
  FFT_LIBS=-L/share/apps/libs/fftw-3.3.2-gcc47/double/lib -lfftw3 -lfftw3_mpi -lfftw3_threads
  MPI_LIBS=-L/share/apps/libs/openmpi47/lib -l:/share/apps/libs/openmpi47/lib/libmpi.so
Please check if this is what you expect.

If any libraries are missing, you may specify a list of directories
to search and retry, as follows:
  ./configure LIBDIRS="list of directories, separated by spaces"

Parallel environment detected successfully.\
Configured for compilation of parallel executables.

For more info, read the ESPRESSO User's Guide (Doc/users-guide.tex).
configure: success

and more...Once it's over do

cd ~/tmp/QE/espresso-5.0
cp * -R /share/apps/QE/
echo 'export PATH=$PATH:/share/apps/QE/bin' >> ~/.bashrc
source ~/.bashrc

8. Testing PW -- STUCK
export LD_LIBRARY_PATH=/share/apps/tools/gcc/gmp/lib:/share/apps/tools/gcc/mpfr/lib:/share/apps/tools/gcc/mpc/lib:/share/apps/libs/openmpi47/lib:/share/apps/libs/openblas47/lib:/share/apps/tools/gcc/gcc47/lib64
cd /share/apps/QE/PW/examples/example02

/share/apps/QE/PW/examples/example02 : starting

This example shows how to use pw.x to compute the equilibrium geometry
of a simple molecule, CO, and of an Al (001) slab.
In the latter case the relaxation is performed in two ways:
1) using the quasi-Newton BFGS algorithm
2) using a damped dynamics algorithm.

  executables directory: /share/apps/QE/bin
  pseudo directory:      /share/apps/QE/pseudo
  temporary directory:   /tmp/QE
  checking that needed directories and files exist...
Downloading O.pz-rrkjus.UPF to /share/apps/QE/pseudo...
Downloading C.pz-rrkjus.UPF to /share/apps/QE/pseudo... done

  running pw.x as: mpirun-gcc-4.7 -n 8  /share/apps/QE/bin/pw.x  

  cleaning /tmp/QE... done
  running the geometry relaxation for CO... from test_input_xml: Empty input file .. stopping
 from test_input_xml: Empty input file .. stopping
 from test_input_xml: Empty input file .. stopping
 from test_input_xml: Empty input file .. stopping
 from test_input_xml: Empty input file .. stopping
 from test_input_xml: Empty input file .. stopping
mpirun-gcc-4.7 noticed that the job aborted, but has no info as to the process
that caused that situation.
Error condition encountered during test: exit status = 2

It's more insidious than that though -- pw.x crashes with "STOP 2" for any type of input.

NOTE: Like on Debian I had segfaults happening when using my own openblas (which works fine with gromacs and nwchem). Hence why I show you how to compile it, yet never use it.

No comments:

Post a Comment