DIGITAL Fortran 90
User Manual for
DIGITAL UNIX Systems

The FLUSH directive identifies synchronization points at which a consistent view of memory is provided. Thread-visible variables are written back to memory at the point at which this directive appears.

The FLUSH directive takes the following form:

c$OMP FLUSH [(list)]

c

Is one of the following: C (or c), !, or * (see Chapter 6).

Rules and Restrictions for the FLUSH Directive

This directive must appear at the precise point in the code at which the synchronization is required. To avoid flushing all variables, specify a comma-separated list of only those variables that need to be flushed using the optional list argument.

The list should contain only named variables. Thread-visible variables include the following data items:

• Globally visible variables (common blocks and modules).
• Local variables that do not have the SAVE attribute but have had their address taken and saved or have had their address passed to another subprogram.
• Local variables that do not have the SAVE attribute that are declared shared in a parallel region within the subprogram.
• Dummy arguments.
• All pointer dereferences.

Modifications to thread-visible variables are visible to all threads after this point. Subsequent reads of thread-visible variables fetch the latest copy of the data.

The FLUSH directive is implied for the following directives:

• BARRIER
• CRITICAL and END CRITICAL
• END DO
• END PARALLEL
• END SECTIONS
• END SINGLE
• ORDERED and END ORDERED

The directive is not implied if a NOWAIT clause is present.

Example

The following example uses the FLUSH directive for point-to-point synchronization between pairs of threads:

c$OMP PARALLEL DEFAULT(PRIVATE) SHARED(ISYNC) IAM = OMP_GET_THREAD_NUM() ISYNC(IAM) = 0 c$OMP BARRIER CALL WORK() C I AM DONE WITH MY WORK, SYNCHRONIZE WITH MY NEIGHBOR ISYNC(IAM) = 1 c$OMP FLUSH(ISYNC) C WAIT TILL NEIGHBOR IS DONE DO WHILE (ISYNC(NEIGH) .EQ. 0) c$OMP FLUSH(ISYNC) END DO c$OMP END PARALLEL

D.1.8 MASTER Directive Construct

The code enclosed within the MASTER and the END MASTER directives is executed by the master thread of the team.

The MASTER directive takes the following form:

c$OMP MASTER block c$OMP END MASTER

c

Is one of the following: C (or c), !, or * (see Chapter 6).

Rules and Restrictions for MASTER and END MASTER Directives

The other threads in the team skip the enclosed section of code and continue execution. There is no implied barrier, neither on entry to nor exit from the master section. This directive has the following restriction:

• The section of code enclosed by the MASTER and END MASTER directives must be a structured block. It is illegal to branch into or out of the block.

Examples

The following example forces the master thread to execute the routines OUTPUT and INPUT:

c$OMP PARALLEL DEFAULT(SHARED) CALL WORK(X) c$OMP MASTER CALL OUTPUT(X) CALL INPUT(Y) c$OMP END MASTER CALL WORK(Y) c$OMP END PARALLEL

D.1.9 ORDERED Directive Construct

The code enclosed within the ORDERED and END ORDERED directives is executed in the order in which iterations would be executed in sequential execution.

The ORDERED directive takes the following form:

c$OMP ORDERED block c$OMP END ORDERED

c

Is one of the following: C (or c), !, or * (see Chapter 6).

Rules and Restrictions for ORDERED and END ORDERED Directives

An ORDERED directive can appear only in the dynamic extent of a DO or PARALLEL DO directive. The DO directive to which the ordered section binds must have the ORDERED clause specified.

One thread is allowed in an ordered section at a time. Threads are allowed to enter in the order of the loop iterations. No thread can enter an ordered section until it can be guaranteed that all previous iterations have completed or will never execute an ordered section. This sequentializes and orders code within ordered sections while allowing code outside the section to run in parallel.

ORDERED sections that bind to different DO directives are independent of each other.

The following restrictions apply to the ORDERED directive:

• The code enclosed by the ORDERED and END ORDERED directives must be a structured block. It is illegal to branch into or out of the block.
• An ORDERED directive cannot bind to a DO directive that does not have the ORDERED clause specified.
• An iteration of a loop using a DO directive must not execute the same ORDERED directive more than once, and it must not execute more than one ORDERED directive.

Example

Ordered sections are useful for sequentially ordering the output from work that is done in parallel. Assuming that a reentrant I/O library exists, the following program prints out the indexes in sequential order:

c$OMP DO ORDERED SCHEDULE(DYNAMIC) DO I=LB,UB,ST CALL WORK(I) END DO . . . SUBROUTINE WORK(K) c$OMP ORDERED WRITE(*,*) K c$OMP END ORDERED

D.1.10 PARALLEL Directive Construct

The PARALLEL directive defines a parallel region.

The PARALLEL directive takes the following form:

c$OMP PARALLEL [clause[[,] clause]...] block c$OMP END PARALLEL

c

Is one of the following: C (or c), !, or * (see Chapter 6).

clause

Is one of the following:

COPYIN (list) DEFAULT ( PRIVATE SHARED NONE ) FIRSTPRIVATE (list) IF (scalar_logical_expression) PRIVATE (list) REDUCTION ( operator intrinsic :list ) SHARED (list)

• COPYIN (list)
  The COPYIN clause applies only to common blocks you declare to be THREADPRIVATE. A COPYIN clause on a parallel region specifies that the data in the master thread of the team be copied to the thread private copies of the common block at the beginning of the parallel region.
  It is not necessary to specify that a whole common block be copied in; you can specify named variables appearing in the THREADPRIVATE common block in the list.
• DEFAULT (
  • PRIVATE
  • SHARED
  • NONE
  )
  The DEFAULT clause allows you to specify a PRIVATE, SHARED, or NONE scope attribute for all variables in the lexical extent of any parallel region. Variables in THREADPRIVATE common blocks are not affected by this clause.
  The PRIVATE, SHARED, and NONE specifications have the following effects:
  • Specifying PRIVATE makes all named objects in the lexical extent of the parallel region, including common block variables but excluding THREADPRIVATE variables, private to a thread as if you explicitly listed each variable in a PRIVATE clause.
  • Specifying SHARED makes all named objects in the lexical extent of the parallel region shared among the threads in a team, as if you explicitly listed each variable in a SHARED clause. In the absence of an explicit DEFAULT clause, the default behavior is the same as if you specified DEFAULT(SHARED).
  • Specifying NONE declares that there is no implicit default as to whether variables are PRIVATE or SHARED. In this case, you must specify the PRIVATE, SHARED, FIRSTPRIVATE, LASTPRIVATE, or REDUCTION attribute of each variable you use in the lexical extent of the parallel region.
  
  You can specify only one DEFAULT clause on a PARALLEL directive. You can exempt variables from a defined default by using the PRIVATE, SHARED, FIRSTPRIVATE, LASTPRIVATE, and REDUCTION clauses.
• FIRSTPRIVATE (list)
  See Section D.1.6.
• IF (scalar_logical-expression)
  When present, the enclosed code region is executed in parallel only if the scalar_logical_expression evaluates to .TRUE.. Otherwise, the parallel region is serialized. The expression must be a scalar DIGITAL Fortran logical expression. In the absence of this clause, the region is executed as if an IF(.TRUE.) clause were specified.
  This clause is evaluated by the master thread before any data scope attributes take effect.
• PRIVATE (list)
  See Section D.1.6.
• REDUCTION (
  • operator
  • intrinsic
  :list )
  See Section D.1.6.
• SHARED (list)
  This clause makes variables that appear in list shared among all the threads in a team. All threads within a team access the same storage area for SHARED data.

Rules and Restrictions for PARALLEL and END PARALLEL Directives

When a thread encounters a parallel region, it creates a team of threads and it becomes the master of the team. The master thread is a member of the team and it has a thread number of 0 within the team. The number of threads in the team is controlled by environment variables and/or library calls.

For more information about environment variables, see Chapter 6. For more information about library routines, see Appendix E.

Once created, the number of threads in the team remains constant for the duration of that parallel region. However, you can explicitly change the number of threads used in the next parallel region by calling the OMP_SET_NUM_THREADS run-time library routine from a serial portion of the program. This routine overrides any value you may have set using the OMP_NUM_THREADS environment variable.

The block denotes a structured block of DIGITAL Fortran statements. It is illegal to branch into or out of the block. The code contained within the dynamic extent of the parallel region is executed on each thread, and the code path can be different for different threads.

The END PARALLEL directive denotes the end of the parallel region. There is an implied barrier at this point. Only the master thread of the team continues execution at the end of a parallel region.

If a thread executing a parallel region encounters another parallel region, it creates a new team and becomes the master of that new team. By default, nested parallel regions are always serialized and executed by a team of one thread.

The following restrictions apply to parallel regions:

• The PARALLEL/END PARALLEL directive pair must appear in the same routine in the executable section of the code.
• The code contained by these two directives must be a structured block. It is illegal to branch into or out of a parallel region.
• Only a single IF clause can appear on the directive.

Examples

You can use the PARALLEL directive in coarse-grain parallel programs. In the following example, each thread in the parallel region decides what part of the global array X upon which to work based on the thread number:

c$OMP PARALLEL DEFAULT(PRIVATE) SHARED(X,NPOINTS) IAM = OMP_GET_THREAD_NUM() NP = OMP_GET_NUM_THREADS() IPOINTS = NPOINTS/NP CALL SUBDOMAIN(X,IAM,IPOINTS) c$OMP END PARALLEL

Once created, the number of threads in the team remains constant for the duration of that parallel region. However, you can explicitly change the number of threads used in the next parallel region by calling the OMP_SET_NUM_THREADS run-time library routine from a serial portion of the program. This routine overrides any value you may have set using the OMP_NUM_THREADS environment variable.

Assuming you had used the OMP_NUM_THREADS environment variable to set the number of threads to 6, you can change the number of threads between parallel regions as follows:

CALL OMP_SET_NUM_THREADS(3) !$OMP PARALLEL . . . !$OMP END PARALLEL CALL OMP_SET_NUM_THREADS(4) !$OMP PARALLEL DO . . . !$OMP END PARALLEL DO

D.1.11 PARALLEL DO Directive Construct

The PARALLEL DO directive provides an abbreviated form of specifying a parallel region that contains a single DO directive.

The PARALLEL DO directive takes the following form:

c$OMP PARALLEL DO [clause[[,] clause]...] do_loop [c$OMP END PARALLEL DO]

c

Is one of the following: C (or c), !, or * (see Chapter 6).

clause

The clause can be one of the clauses accepted by the PARALLEL or DO directives. See Sections D.1.10 and Section D.1.6\value).

Rules and Restrictions for PARALLEL DO and END PARALLEL DO Directives

If the END PARALLEL DO directive is not specified, the PARALLEL DO is assumed to end with the DO loop that immediately follows the PARALLEL DO directive. If used, the END PARALLEL DO directive must appear immediately after the end of the DO loop.

The semantics are identical to explicitly specifying a PARALLEL directive immediately followed by a DO directive.

Examples

In the following example, the loop iteration variable is private by default and it is not necessary to declare it explicitly. The END PARALLEL DO directive is optional:

c$OMP PARALLEL DO DO I=1,N B(I) = (A(I) + A(I-1)) / 2.0 END DO c$OMP END PARALLEL DO

The following example shows how to use the REDUCTION clause:

c$OMP PARALLEL DO DEFAULT(PRIVATE) REDUCTION(+: A,B) DO I=1,N CALL WORK(ALOCAL,BLOCAL) A = A + ALOCAL B = B + BLOCAL END DO c$OMP END PARALLEL DO

D.1.12 PARALLEL SECTIONS Directive Construct

The PARALLEL SECTIONS directive provides an abbreviated form of specifying a parallel region that contains a single SECTIONS directive. The semantics are identical to explicitly specifying a PARALLEL directive immediately followed by a SECTIONS directive.

The PARALLEL SECTIONS directive takes the following form:

c$OMP PARALLEL SECTIONS [clause[[,] clause]...] [c$OMP SECTION ] block [c$OMP SECTION block] . . . c$OMP END PARALLEL SECTIONS

c

Is one of the following: C (or c), !, or * (see Chapter 6).

clause

The clause can be one of the clauses accepted by the PARALLEL or SECTIONS directives. See Sections D.1.10 and D.1.13.

Rules and Restrictions for PARALLEL SECTIONS and END PARALLEL SECTIONS Directives

The last section ends at the END PARALLEL SECTIONS directive.

Example

In the following example, subroutines XAXIS, YAXIS, and ZAXIS can be executed concurrently:

c$OMP PARALLEL SECTIONS c$OMP SECTION CALL XAXIS c$OMP SECTION CALL YAXIS c$OMP SECTION CALL ZAXIS c$OMP END PARALLEL SECTIONS

D.1.13 SECTIONS Directive Construct

The SECTIONS directive is a noniterative worksharing construct specifying that the enclosed sections of code must be divided among threads in the team. Each section is executed once by a thread in the team.

A SECTIONS directive takes the following form:

c$OMP SECTIONS [clause[[,] clause]...] [c$OMP SECTION] block [c$OMP SECTION block] . . . c$OMP END SECTIONS [NOWAIT]

c

Is one of the following: C (or c), !, or * (see Chapter 6).

clause

Is one of the following:

FIRSTPRIVATE(list) LASTPRIVATE(list) PRIVATE(list) REDUCTION( operator intrinsic :list )

• FIRSTPRIVATE(list)
  See Section D.1.6.
• LASTPRIVATE(list)
  See Section D.1.6.
• PRIVATE(list)
  See Section D.1.6.
• REDUCTION(
  • operator
  • intrinsic
  :list )
  See Section D.1.6.

Rules and Restrictions for SECTIONS and END SECTIONS Directives

Each section of code is preceded by a SECTION directive, although the SECTION directive is optional for the first section. The SECTION directives must appear within the lexical extent of the SECTIONS/END SECTIONS directive pair. The last section ends at the END SECTIONS directive. Threads that complete execution of their SECTIONs encounter an implied barrier at the END SECTIONS directive unless NOWAIT is specified.

The following restrictions apply to the SECTIONS directive:

• The code enclosed in a SECTIONS/END SECTIONS directive pair must be a structured block. In addition, each constituent section must also be a structured block. It is illegal to branch into or out of the constituent section blocks.
• It is illegal for a SECTION directive to be outside the lexical extent of the SECTIONS/END SECTIONS directive pair.
• SECTIONS directives must be encountered by all threads in a team or by none at all.
• SECTIONS directives must be encountered in the same order by all threads in a team.

Example

In the following example, subroutines XAXIS, YAXIS, and ZAXIS can be executed concurrently:

c$OMP PARALLEL c$OMP SECTIONS c$OMP SECTION CALL XAXIS c$OMP SECTION CALL YAXIS c$OMP SECTION CALL ZAXIS c$OMP END SECTIONS c$OMP END PARALLEL

D.1.14 SINGLE Directive Construct

The SINGLE directive specifies that the enclosed code is to be executed by only one thread in the team. Threads in the team that are not executing this directive wait at the END SINGLE directive unless NOWAIT is specified.

The SINGLE directive takes the following form:

c$OMP SINGLE [clause[[,] clause]...] block c$OMP END SINGLE [NOWAIT]

c

Is one of the following: C (or c), !, or * (see Chapter 6).

clause

Is one of the following:

FIRSTPRIVATE(list) PRIVATE(list)

• FIRSTPRIVATE(list)
  See Section D.1.6.
• PRIVATE(list)
  See Section D.1.6.

Rules and Restrictions for SINGLE and END SINGLE Directives

The following restrictions apply to a SINGLE directive:

• The code enclosed in a SINGLE/END SINGLE directive pair must be a structured block. It is illegal to branch into or out of the block.
• SINGLE directives must be encountered by all threads in a team or by none at all.
• SINGLE directives must be encountered in the same order by all threads in a team.

Example

In the following example, the first thread that encounters the SINGLE directive executes subroutines OUTPUT and INPUT. You must not make any assumptions as to which thread executes the SINGLE section. All other threads skip the SINGLE section and stop at the barrier at the END SINGLE construct. If other threads can proceed without waiting for the thread executing the SINGLE section, specify NOWAIT on the END SINGLE directive:

c$OMP PARALLEL DEFAULT(SHARED) CALL WORK(X) c$OMP BARRIER c$OMP SINGLE CALL OUTPUT(X) CALL INPUT(Y) c$OMP END SINGLE CALL WORK(Y) c$OMP END PARALLEL