CGPACK progress, NOV-2016


next | UP | previous

7-NOV-2016: A report on TAU and CrayPAT

paper.pdf - profiling of CGPACK and CGPACK+ParaFEM multi-scale models with TAU and CrayPAT.

8-NOV-2016: Checking new NetCDF writer

Luis has written unit test testACF.f90 which writes CA data in NetCDF format. In addition this test also does timing. The user prepares several directories with different lsf settings.

Below is the output file written in NetCDF read into ParaView. The cells are initialised with the image number. The code was run on 192 images, so the colour bar shows cubes with colours from dark blue (image 1) to red (image 192). The file was saved with *.ncdf extension. In was read as ParaView "netCDF files generic and CF conventions".

CA array saved in NetCDF format read into ParaView

Since this seems to work, cut release 2.7.

ncdump is a useful utility. It can convert a binary NetCDF file into plain text. This is particularly useful for extracting metadata.

$ ncdump -cs netcdf.dat 
netcdf netcdf {
        x = 464 ;
        y = 930 ;
        z = 1392 ;
        int data(z, y, x) ;
                data:_Storage = "contiguous" ;
                data:_Endianness = "little" ;
                data:_NoFill = "true" ;

// global attributes:
                :_Format = "netCDF-4" ;

9-NOV-2016: NetCDF timings

testACF.f90 has been updated to automatically create dirs with different lfs stripe counts and sizes. The following values have been studied:

  character( len=maxlen), dimension(9) :: stripe_count = (/            &
    "-c-1 ", "-c0  ", "-c1  ", "-c4  ", "-c8  ",                       &
    "-c16 ", "-c20 ", "-c32 ", "-c40 " /)
  character( len=maxlen), dimension(7) :: stripe_size = (/             &
    "-S1m ", "-S2m ", "-S4m ", "-S8m ", "-S16m",                       &
    "-S32m", "-S64m" /)

Files are saved in directores with names formed by stripe count and size. Each file has about 10^9 cells, i.e. about 4.5GB.

Data from 8 Archer nodes, sorted by IO rate in descending order. Pure MPI/IO peaks at 4GB/s. NetCDF gives only just above 1GB/s.

13-NOV-2016: Luis added an HDF5 writer

Luis added an HDF5 writer in module cgca_m2hdf5.f90. I updated test testACF.f90 to dump the model also in HDF5.

HDF5 file cannot be read directly into ParaView, because the metadata contained in HDF5 file does not fully describe it. I had to write a simple xdmf metadata file for it: hdf5.xdmf. The xdmf file is given to ParaView.

The necessary metadata can be extracted from the hdf5 file with h5ls and h5dump h5dump utilities:

$ h5ls hdf5.dat 
IntArray                 Dataset {1392, 930, 930}

$ h5dump -H hdf5.dat 
HDF5 "hdf5.dat" {
GROUP "/" {
   DATASET "IntArray" {
      DATASPACE  SIMPLE { ( 1392, 930, 930 ) / ( 1392, 930, 930 ) }

Below is the CA brick where cell values are image numbers. This was run on 8 ARcher cores (480 images).

CA array saved in HDF5 format read into ParaView

validate this page

next | UP | previous