malloc(): unaligned tcache chunk detected. Has anyone faced this before for MPI fortran programs?

Question

I have an MPI program, where I face the "malloc(): unaligned tcache chunk detected" error if I run it on one processor, but not on 8 processors. The memory allocation looks like this:


  ALLOCATE(XPOINTS((Npx+1)))
  IF(MY_RANK .eq. 0) WRITE(*,*)  "TESTING"
  ALLOCATE(YPOINTS((Npy+1)))
  ALLOCATE(ZPOINTS((Npz+1)))
  ALLOCATE(x_GLBL((1-Ngl):(Nx_glbl+Ngl)))
  ALLOCATE(y_GLBL((1-Ngl):(Ny_glbl+Ngl)))
  ALLOCATE(z_GLBL((1-Ngl):(Nz_glbl+Ngl)))

Note, that I have verified all the numbers for allocation are integers. This is the error that I am seeing:

 TESTING
malloc(): unaligned tcache chunk detected
malloc(): unaligned tcache chunk detected

Program received signal SIGABRT: Process abort signal.

Program received signal SIGABRT: Process abort signal.

Backtrace for this error:

Backtrace for this error:
#0  0x7f2145348960 in ???
#1  0x7f2145347ac5 in ???
#2  0x7f214513e51f in ???
        at ./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c:0
#3  0x7f21451929fc in __pthread_kill_implementation
        at ./nptl/pthread_kill.c:44
#4  0x7f21451929fc in __pthread_kill_internal
        at ./nptl/pthread_kill.c:78
#5  0x7f21451929fc in __GI___pthread_kill
        at ./nptl/pthread_kill.c:89
#6  0x7f214513e475 in __GI_raise
        at ../sysdeps/posix/raise.c:26
#7  0x7f21451247f2 in __GI_abort
        at ./stdlib/abort.c:79
#8  0x7f2145185675 in __libc_message
        at ../sysdeps/posix/libc_fatal.c:155
#9  0x7f214519ccfb in malloc_printerr
        at ./malloc/malloc.c:5664
#10  0x7f21451a13db in tcache_get
        at ./malloc/malloc.c:3195
#11  0x7f21451a13db in __GI___libc_malloc
        at ./malloc/malloc.c:3313
#12  0x55ecaeda5ab3 in ???
#13  0x55ecaed90452 in ???
#14  0x55ecaed902ee in ???
#15  0x7f2145125d8f in __libc_start_call_main
        at ../sysdeps/nptl/libc_start_call_main.h:58
#16  0x7f2145125e3f in __libc_start_main_impl
        at ../csu/libc-start.c:392
#17  0x55ecaed90324 in ???
#18  0xffffffffffffffff in ???
#0  0x7efe26f48960 in ???
#1  0x7efe26f47ac5 in ???
#2  0x7efe26d3e51f in ???
        at ./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c:0
#3  0x7efe26d929fc in __pthread_kill_implementation
        at ./nptl/pthread_kill.c:44
#4  0x7efe26d929fc in __pthread_kill_internal
        at ./nptl/pthread_kill.c:78
#5  0x7efe26d929fc in __GI___pthread_kill
        at ./nptl/pthread_kill.c:89
#6  0x7efe26d3e475 in __GI_raise
        at ../sysdeps/posix/raise.c:26
#7  0x7efe26d247f2 in __GI_abort
        at ./stdlib/abort.c:79
#8  0x7efe26d85675 in __libc_message
        at ../sysdeps/posix/libc_fatal.c:155
#9  0x7efe26d9ccfb in malloc_printerr
        at ./malloc/malloc.c:5664
#10  0x7efe26da13db in tcache_get
        at ./malloc/malloc.c:3195
#11  0x7efe26da13db in __GI___libc_malloc
        at ./malloc/malloc.c:3313
#12  0x55fa223ddab3 in ???
#13  0x55fa223c8452 in ???
#14  0x55fa223c82ee in ???
#15  0x7efe26d25d8f in __libc_start_call_main
        at ../sysdeps/nptl/libc_start_call_main.h:58
#16  0x7efe26d25e3f in __libc_start_main_impl
        at ../csu/libc-start.c:392
#17  0x55fa223c8324 in ???
#18  0xffffffffffffffff in ???

Has anyone faced this before? I tried everything and cant figure out why it doesnt work on less than 8 processors. Tried it with both Intel and GNU fortran. Is this a problem specific to my laptop?

I tried it with both Intel and GNU compilers. It works for 8 processors but not for 1 processor.

Edit: I am unable to reproduce this error in a simpler program, so I am attaching the git hub repo: https://github.com/SahajSJain/MyPoisonX.git Edit: I was finally able to recreate the error:

PROGRAM create_cart_coords
use, intrinsic :: iso_fortran_env
use mpi
implicit none
integer process_Rank, size_Of_Cluster, ierror, tag, Nx, Ny, Nz, Ngl
real(kind=real64), allocatable, dimension(:) :: array_x, array_y, array_z
real(kind=real64), allocatable, dimension(:,:,:) :: ALPHA, BETA, GAMMA

INTEGER, DIMENSION(3) :: NumProcArr,PeriodicArr, myCOORDS
INTEGER :: MaxDims, CommCart, reorder, myCartRank
INTEGER :: myWest, myEast, mySouth, myNorth, myBack, myFront
call MPI_INIT(ierror)
call MPI_COMM_SIZE(MPI_COMM_WORLD, size_Of_Cluster, ierror)
call MPI_COMM_RANK(MPI_COMM_WORLD, process_Rank, ierror)
NumProcArr=(/1,1,1/);
PeriodicArr=(/0,0,0/)
reorder=1;
MaxDims=3
print *, 'Hello World from process: ', process_Rank, 'of ', size_Of_Cluster
!! do cartesian mapping
call MPI_Cart_create(MPI_COMM_WORLD, MAXDIMS, NumProcArr,PeriodicArr, reorder, CommCart, ierror)
call MPI_Comm_rank(CommCart,myCartRank ,ierror);
call MPI_Cart_coords(CommCart, myCartRank, MAXDIMS, myCOORDS);
WRITE(*,*) "I am MPI Process" , myCartRank," out of ", size_Of_Cluster,", and I am located at",myCOORDS
CALL MPI_CART_SHIFT(CommCart, 0, 1, myWest, myEast, ierror)
CALL MPI_CART_SHIFT(CommCart, 1, 1, mySouth, myNorth, ierror)
CALL MPI_CART_SHIFT(CommCart, 2, 1, myBack, myFront, ierror)
CALL MPI_Barrier(CommCart,ierror)
Nx=50;
Ny=50;
Nz=50;
Ngl=2;
if(process_Rank .eq. 0) WRITE(*,*) "STARTING ALLOCATION"
ALLOCATE(array_x((1-Ngl):(Nx+Ngl)));
if(process_Rank .eq. 0) WRITE(*,*) "ALLOCATED ARRAY_X"
ALLOCATE(array_y((1-Ngl):(Ny+Ngl)));
if(process_Rank .eq. 0) WRITE(*,*) "ALLOCATED ARRAY_y"
ALLOCATE(array_z((1-Ngl):(Nz+Ngl)));
if(process_Rank .eq. 0) WRITE(*,*) "ALLOCATED ARRAY_Z"
ALLOCATE(ALPHA((1-Ngl):(Nx+Ngl),(1-Ngl):(Ny+Ngl),(1-Ngl):(Nz+Ngl)));
if(process_Rank .eq. 0) WRITE(*,*) "ALLOCATED ALPHA"
ALLOCATE(BETA((1-Ngl):(Nx+Ngl),(1-Ngl):(Ny+Ngl),(1-Ngl):(Nz+Ngl)));
if(process_Rank .eq. 0) WRITE(*,*) "ALLOCATED BETA"
ALLOCATE(GAMMA((1-Ngl):(Nx+Ngl),(1-Ngl):(Ny+Ngl),(1-Ngl):(Nz+Ngl)));
if(process_Rank .eq. 0) WRITE(*,*) "ALLOCATED GAMMA"

call MPI_FINALIZE(ierror)
END PROGRAM

Joachim · Accepted Answer · 2025-01-17 16:53:03Z

2

The message malloc(): unaligned tcache chunk detected is an error message from the underlying implementation of allocate. In your case, the implementation of malloc seems to store additional meta information about the allocation block next to the heap allocation. During an allocation, malloc detects that this meta data was corrupted, which is typically caused by out-of-bounds write to another allocation.

In general, AddressSanitizer and valgrind are tools to detect such out of bounds accesses during execution. As commented by @IanBush, for Fortran adding -fcheck=all is sufficient to detect these errors.

I tried to compile your code with gfortran and OpenMPI. The compiler complains that the calls to MPI_Cart_create and MPI_Cart_coords do not match with the declaration. For MPI_Cart_create, the PeriodicArr argument is expected to be LOGICAL(*). The call to MPI_Cart_coords is missing the ierror argument.

After fixing these errors to successfully compile, I cannot reproduce your error on my system, but the missing ierror argument might already explain the issue.

The execution with mpirun -np 2 ./MyPoisonX reports:

At line 199 of file CODE.SETUP_FIELD_VARIABLES.F90
Fortran runtime error: Index '27' of dimension 1 of array 'dxinv' above upper bound of 26

Which is a result from -fcheck=all analysis.

To use AddressSanitizer, you would add -fsanitize=address to the CFLAGS and LFLAGS in your Makefile.

Afterwards, execution with mpirun -np 2 env ASAN_OPTIONS="detect_leaks=0" ./MyPoisonX would report errors detected by ASan. I suggest disabling leak checks to avoid flooding the screen with tons of MPI-related memory leaks.

For this specific code, ASan does not report OOB errors, while the Fortran-specific analysis reports errors. The reason is that ASan cannot detect index overflows, if they result in valid accesses to array memory, i.e. into the begin of the next column.

edited Jan 17 at 16:53

answered Jan 16 at 10:05

Joachim

1,3631 gold badge8 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Ian Bush Jan 16 at 10:17

This is Fortran, so if there is an out of bounds access simply -fcheck=all on the compile line should be sufficient to find it

Ian Bush Jan 16 at 13:46

But admittedly if there is a mistake in the arguments to a MPI routine -fsanitize=address may be required - but I would use -fcheck=all first and only then get out the chainsaw if required.

Sahaj Jain Jan 16 at 15:20

Hi! Thank you so much! This seems to have fixed it. I think the problem might have been that I was looking at the C subroutines for MPI and might have forgotten that I need to add ierror for Fortran. Strange it seemed to work for 8 processors though.

Joachim Jan 17 at 17:07

@IanBush I integrated your comment and added a bit more context for the use of both flags. I also tried to explain why the error is reported by -fcheck=all and not by ASan

Gilles Gouaillardet Jan 18 at 2:29

This is one more reason to modernize your code and move to use mpi_f08 if your compiler and library support it.

|

Collectives™ on Stack Overflow

malloc(): unaligned tcache chunk detected. Has anyone faced this before for MPI fortran programs?

1 Answer 1

6 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Related