Caffeine

CoArray Fortran Framework of Efficient Interfaces to Network Environments

Caffeine is a parallel runtime library that aims to support Fortran compilers
with a programming-model-agnostic application interface to various
communication libraries. Current work is on supporting the Parallel Runtime
Interface for Fortran (PRIF) with the GASNet-EX exascale-ready networking
middleware. The Caffeine team anticipates collaborating on separate alternative
PRIF implementations using other communication substrates such as the Message
Passing Interface (MPI).

Caffeine system stack diagram

Statement of need

The Fortran programming language standard added features supporting
single-program, multiple-data (SPMD) parallel programming and loop
parallelism beginning with Fortran 2008. In Fortran, SPMD programming
involves the creation of a fixed number of images (instances) of a
program that execute asynchronously in shared or distributed memory, except
where a program uses specific synchronization mechanisms. Fortran's
coarray feature provides distributed data structures that offer a subscripted,
multidimensional array notation defining a partitioned global address space
(PGAS). One image can use a coindexed notation to perform one-sided access
of coarray data associated with another image.

Fortran 2018 greatly expanded this feature set to include concepts such as
teams (groupings) of images, events (counting semaphores), collective
subroutines and failed-image detection (fault tolerance). Fortran 2023 provided
additional, minor multi-image extensions, including notified remote data access.

Several popular Fortran compilers, including LLVM Flang
and LFortran, currently
lack complete support for multi-image parallel execution. These features are a mandatory
part of Fortran, and thus are an important part of reaching full compliance with
the 2008, 2018, or 2023 revisions of the Fortran standard.
The latest LLVM Flang 22 release adds experimental support for a
meaningful subset
of multi-image Fortran features using PRIF and Caffeine.
For more details, see LLVM-HPC2025 paper below.

Caffeine provides a portable, high-performance and open-source parallel
runtime library that compilers can target in code generation as part of
their solution to support Fortran's multi-image parallel features.

Prerequisites & Dependencies

Build prerequisites

The install.sh script uses the following packages:

Fortran and C compilers
- We regularly test with: LLVM Flang versions 19:22 and GNU Fortran versions 13:15
Fortran package manager fpm
pkg-config
realpath
make
git
curl

The script will invoke these if present in a user's PATH.
If not present, the script will ask permission to use Homebrew to install the relevant package
or, in some cases, ask the user to install the package.

Build dependencies

Caffeine also depends on the following packages that will be automatically installed as part
of the build process.

GASNet-EX exascale networking middleware
assert
julienne

Caffeine leverages the following non-parallel features of Fortran to simplify the writing of a portable, compact runtime-library that supports Fortran's parallel features:

Feature	Introduced in
The `iso_c_binding` module	Fortran 2003
The `contiguous` attribute	Fortran 2008
Submodule support [1]	Fortran 2008
The `ISO_Fortran_binding.h` C header file	Fortran 2018
Assumed-type dummy arguments: `type(*)`	Fortran 2018
Assumed-rank dummy arguments: `array(..)`	Fortran 2018

[1] This feature simplifies development but is not essential to the package

Download, build, and run an example

Here is an outline of the basic commands used to build Caffeine and run an example:

git clone https://github.com/BerkeleyLab/caffeine.git
cd caffeine
env FC=<Fortran-compiler> CC=<C-compiler> CXX=<C++-compiler> ./install.sh <options>
env CAF_IMAGES=8 ./run-fpm.sh run --example hello

If $FC or $CC are unset, then install.sh will look for LLVM-based compilers in
the PATH, and failing that, offer to install such compilers using Homebrew.

The selected compilers MUST be "compatible": for the best experience you are
HIGHLY recommended to specify the language frontends provided by a single version
of a given compiler suite installation. The C++ compiler is optional for
single-node deployments (and can be disabled using command-line option --without-cxx),
but C++ is required for some network backends.

Note for macOS/Homebrew users: Homebrew may be used to install
LLVM flang for use with Caffeine, however
by default Homebrew will not replace the incompatible Apple-provided clang in your PATH.
Such users are recommended to use Homebrew's matching LLVM C/C++ compilers
(usually installed in /opt/homebrew/opt/llvm/bin)
to ensure that a matching set of LLVM compilers are used.

The install.sh recognizes a number of command-line options and environment variables to
customize behavior for your system. See the output of ./install.sh --help for full documentation,
including options for how to build for a distributed-memory platform or with thread-safety.

Example Usage

The Caffeine parallel runtime is intended as an embedded compilation target
library, to provide multi-image parallel runtime support to a Fortran compiler.
As such, real usage of Caffeine is specific to the host Fortran compiler, and
one should consult compiler-provided documentation regarding the use of Caffeine
to back multi-image features.

However we provide an example hello world program,
written in Fortran, simulating the PRIF calls that a theoretical
source-to-source Fortran compiler might generate for a simple program written
using Fortran's multi-image features to print a message from each image.

Run tests

After installation, one can optionally issue the following command to run
Caffeine's correctness unit tests to exercise the PRIF subroutines:

./run-fpm.sh test

Note that some unit tests are conditionally skipped based on platform and configuration details.

Recognized Environment Variables

The following environment variables control the execution of the fpm-driven Caffeine unit test suite:

CAF_IMAGES: integer that indicates the number of images to run
SUBJOB_PREFIX: command prefix to use for recursive fpm invocations in the test suite.
Set SUBJOB_PREFIX=skip to disable such invocations (recommended for distributed-memory systems).

The following environment variables control the behavior of the Caffeine library:

CAF_HEAP_SIZE=128MB: set the size of the shared-memory heap used for coarray storage, defaults to 128 MiB
CAF_COMP_FRAC=0.10: set the fraction of the shared-memory heap reserved for non-symmetric allocation, defaults to 10%

Caffeine is built atop the GASNet-EX exascale networking middleware, which has its own
set of environment variable knobs to control network-level behavior.
Here are a few of the most useful GASNet knobs:

GASNET_VERBOSEENV=1: enable console output of all the envvar settings affecting GASNet operation
GASNET_SPAWN_VERBOSE=1: enable verbose console output of parallel job-spawning steps
GASNET_BACKTRACE=1: enable automatic backtrace upon fatal errors
GASNET_SSH_SERVERS="host1 host2": space-deliminted list of hostnames for distributed-memory job launch using the ssh-spawner

See GASNet documentation for full details on all settings.

PRIF Implementation Status

Caffeine is an implementation of the Parallel Runtime Interface for Fortran (PRIF)

For details on the PRIF features that are implemented, please see the Implementation Status doc.

Publications

Funding

The Computer Languages and Systems Software (CLaSS) Group at Berkeley Lab has developed Caffeine
on funding from the Exascale Computing Project (ECP)
and the Stewardship for Programming Systems and Tools (S4PST) project,
part of the Consortium for the Advancement of Scientific Software (CASS).