Lib-DVM interface description (contents) Part 1
(1-5
)
Part 2
(6-7)
Part 3
(8-11)
Part 4
(12-13)
Part 5
(14-15)
Part 6
(16-18)
Part 7
(19)
created: february, 2001 - last edited 03.05.01 -

6 Distributed array creating and deleting

6.1 Creating distributed array

long crtda_ ( long
long
void
long
long
long
long
long
long
long
ArrayHeader[],
*ExtHdrSignPtr,
*BasePtr,
*RankPtr,
*TypeSizePtr,
SizeArray[],
*StaticSignPtr,
*ReDistrParPtr,
LowShdWidthArray[],
HiShdWidthArray[] );
     
ArrayHeader - header of the array to be created.
*ExtHdrSignPtr - flag of extended header.
BasePtr - base pointer, used to access to the distributed array elements.
*RankPtr - rank of the created array.
*TypeSizePtr - size of the array element (in bytes).
SizeArray - array, which i-th element is a size of the (i+1)-th dimension of the created (0 i *RankPtr-1).
*StaticSignPtr - the flag of the static array creation.
*ReDistrParPtr - the permission flag of using of created distributed array as a parameter in the function redis_ and realn_ (see sections 5.2 and 7.3).
LowShdWidthArray - array, which i-th element is a width of the low shadow edge of the (i+1)-th dimension of the distributed array.
HiShdWidthArray - array, which i-th element is a width of the high shadow edge of the (i+1)-th dimension of the distributed array.

The creation of a distributed array using function crtda_ means only an initialization of the internal system structures describing the array. The memory for the distributed array elements is allocated only after mapping this array on an abstract machine by the function align_. (In another words, memory is allocated after evaluating for each processor abstract machine it is mapped onto, a local part of distributed array according to its distribution, see section 7.1). An access to array elements is performed through Run-Time System functions (see section 13).

A header of the created array is an array of *RankPtr+1 (if *ExtHdrSignPtr = 0) or *RankPtr*2+2 (if *ExtHdrSignPtr = 1) elements of "long" type. The user program has to allocate this header in memory (whether statically or dynamically). Run-Time System only initializes this header. The main purpose of this header is to provide an effective access to the elements of the local part of the distributed array.

The base pointer BasePtr has to point to any variable of the same type as that of distributed array element. The value of this variable is not essential because the base pointer is only used to access to the elements from the local part of the distributed array.

The structure and usage of distributed array header (jointly with base pointer) to access to the array local part are considered in section 13.4.

If the flag *StaticSignPtr of the static array creation is not equal to zero, then the created distributed array is not deleted automatically when the control exits from the current program block (see section 8). Such array has to be deleted explicitly using the function delda_.

The parameter *ReDistrParPtr may have the following values:

0 - the array can not be an argument of the function redis_ and can not be redistributed by the function realn_;
1 - the array can be an argument of the function redis_, but can not be redistributed by the function realn_;
2 - the array can be redistributed by the function realn_, but this array can not be used in defining the abstract machine representation using the function redis_;
3 - the array can be used as a reference to the abstract machine representation in the function redis_, and the array can be redistributed by the function realn_.

The number of the parameters defined in the arrays LowShdWidthArray and HiShdWidthArray (that is the width of low and high shadow edges) has to be equal to the created array rank. See section 12 for more detail description of the shadow edges of the distributed array local parts. Note, that the shadow edge width can be equal to zero.

The function returns zero.

6.2 Deleting distributed array

long delda_(long ArrayHeader[]);

ArrayHeader – the header of the array to be deleted.

The function deletes the distributed array created by the function crtda_. After array deleting the user program can use distributed array header for any goal.

The distributed array can be deleted by delda_ function only if it was created in the current subtask and in the current program block (or its sub-block) (see sections 8 and 10). All edge groups, in which the array was included, (see section 12) will be deleted also.

A distributed array can be deleted also by the function delobj_ (see section 17.5).

The function returns zero.

6.3 Creating additional header of distributed array

long addhdr_ ( long
long
NewArrayHeader[],
ArrayHeader[] );
 
NewArrayHeader - header to be created.
ArrayHeader - one of the existing headers.

The function creates the distributed array header, which further will be used equitably with existing headers. In particular it can be used to create other additional headers.

When the distributed array is aligned by functions align_ and malign_ (see section 7) all its headers are initialized. The array realigning by the functions realn_, mrealn_, redis_ and mredis_ (see sections 5 and 7) also causes all its headers updating.

The function returns zero.

6.4 Deleting distributed array header

long delhdr_ (long ArrayHeader[]);

ArrayHeader - header to be deleted.

The function deletes the header, created by addhdr_ function or specified in crtda_ funcion call. The only header of distributed array can't be deleted.

The function returns zero.

7 Mapping distributed array

7.1 Aligning distributed array

long align_ ( long
PatternRef
long
long
long
ArrayHeader[],
*PatternRefPtr,
AxisArray[],
CoeffArray[],
ConstArray[] );
 
ArrayHeader - the header of the distributed array to be aligned.
*PatternRefPtr - reference to the alignment pattern.
AxisArray - array, which j-th element is a dimension number of the distributed array used in the linear alignment rule for the pattern (j+1)-th dimension.
CoeffArray - array, which j-th element is a coefficient for distributed array index variable used in the linear alignment rule for the pattern (j+1)-th dimension.
ConstArray - array, which j-th element is a constant used in the linear alignment rule for the pattern (j+1)-th dimension.

The function align_ defines the location of the specified distributed array in the assigned abstract machine representation space (in other words the function aligns the distributed array). Besides, the elements of the array are distributed among the child abstract machines incorporating in the representation. The abstract machine representation is defined by means of the pattern, which can be as the representation itself, or a distributed array (this is called "indirect mapping"). In the last case the reference to the pattern is a first word of distributed array header.

Using the distributed array as a pattern is allowed only if this array has already been mapped on an abstract machine representation. Moreover, using an abstract machine representation as a pattern is allowed only if this representation has been mapped on a processor system. So the function align_ only distributes elements of the aligned distributed array among processors from this processor system.

Aligning pattern can be mapped (as final result) on the current processor system or its subsystem only.

It is not allowed to map already mapped distributed array.

Consider distributed array alignment in detail. Let F be a multifunctional with the domain of definition in a space of indexes of the aligned array and with the image in the space of indexes of the pattern:

F( (I1, ... , Ii, ... , In) ) = F1(I1, ... , Ii, ... , In)
. . . . . . . . . . . . .. .
F
j(I1, ... , Ii, ... , In)
. . . . . . . . . . . . .. .
F
m(I1, ... , Ii, ... , In)

where:

- is a symbol of the Cartesian product;
n - rank of the distributed array;
m - rank of the pattern;
Ii - index variable of the i-th dimension of the aligned array;
Fj - multifunction, which range of value is a range of index variable of pattern j-th dimension.

An alignment of the distributed array with the pattern is a defining the F multifunction. This defining designates that an element (I1, ... , In) of an aligned array has to be located on the processor if and only if some pattern elements from F((I1, ... , In)) set are mapped on this processor. The set F((I1, ... , In)) is an image-set, and components of this set are vectors of space of indexes of the pattern.

The F1, ... , Fm functions are named "coordinate alignment rules". Run-Time System provides the following set of alignment rules.

1. Fj(I1, ... , In) = {Aj*Ik + Bj} , where:

k=f(j) - dimension number of an aligned array (1 k n, f(j1) f(j2) when j1 j2);
Aj - integer;
Bj - nonnegative integer.

This alignment rule means that for each element (I1, … , In) of index space of the aligned array the corresponding image-set contains one element – Aj*Ik + Bj. This element belongs to the value range of the index variable of the pattern j-th dimension.

Note, that Aj and Bj have to meet the following requirements:

0 Bj MAXj; and 0 Aj*MAXk + Bj MAXj , where:

MAXj - maximum of the index variable of the j-th dimension;
MAXk - maximum of the index variable of the aligned array k-th dimension.

2. Fj(I1, ... , In) = {q Mj: 0 q MAXj} , where:

Mj - range of values of the index value of the pattern j-th dimension;
MAXj - maximum of the index variable of the pattern j-th.

This mapping rule means that for each element (I1, … , In) of index space of the aligned array a corresponding image-set, consists of whole range of values of index variable of the pattern j-th dimension. In such a case, the symbol "*" ("any of the admissible") is usually used.

Examples.

  1. Alignment rule F( (I1,I2) ) = {I1} {I2} means that element of two-dimensional array has to be mapped onto any processor, an element of two-dimensional pattern is mapped on, if the indexes of both elements are equal.
  2. Alignment rule F( (I1,I2) ) = {*} {I1+5} {*} means that the element of two-dimensional array has to be mapped onto any processor, an element of three-dimensional pattern is mapped on, if the index of the pattern second dimension is equal to the index of the aligned array first dimension plus 5. The index of the aligned array second dimension and the indexes of the pattern first and third dimension are not considered.
  3. Alignment rule F( (I1,I2,I3) ) = {*} {*} means, that each element of three-dimensional array has to be mapped onto each processor, any element of two-dimension pattern is mapped on. The indexes of the aligned array and of the pattern are not considered.
  4. Alignment rule F( (I1,I2) ) = {0} {1} {2} means, that an element of two-dimensional array has to be mapped onto each processor, the element (0,1,2) of three-dimensional pattern is mapped on. The indexes of aligned array are not considered.

Defining the aligned array distribution among pattern space (that is the defining F1, ... , Fm functions) it is necessary to take into account that all elements of the distributed array have to be inside the pattern space. Observance of correct mapping on each stage of a chain of alignments guarantees the correct final distribution of array elements among the processors.

Note that if an array is used as a pattern, then after realignment of this array by the function realn_ (see section 7.3), the array aligned with this pattern keeps its location.

When the function align_ is called the parameters of the alignment rule Fj(I1, ... , In)= {Aj*Ik + Bj} for the j-th pattern dimension have to be defined as follows:

AxisArray[j-1] contains value k;
CoeffArray[j-1] contains value Aj;
ConstArray[j-1] contains value Bj.

To define alignment rule Fj(I1, ... , In) with image in set of all values of the index variable of j-th dimension of template for any I1, ... , In, the value AxisArray[j-1] (k value) has to be set to -1. The values CoeffArray[j-1] and ConstArray[j-1] are irrelevant in this case.

The number of the alignment rules has to be equal to the rank of the pattern, when the function align_ is called.

The function returns non-zero value, if mapped array has a local part on the current processor, and zero otherwise.

Note 1. The distributed array mapping, specified for function align_, may be implemented only in the case, if for all the processors, at least one array element is mapped onto, a local size of each array dimension, obtained as a result of mapping, is not less than widths of high and low shadow edges of the dimension.

Note 2. If DisArrayFill parameter of Run-Time system startup is non-zero (sysdebug.* files of base parameter sets), mapped distributed array will be cleared by "zero" byte, i.e. by the byte with all zero bits). If DisArrayFill parameter is zero, the mapped array contents is undefined.

7.2 Alignments superposition

Let the function G defines the location of the n-dimensional array in the space of the r-dimensional pattern; and let the function P defines the location of the r-dimensional array in the space of the m-dimensional pattern:

G((I1, ... , Ii, ... , In)) = G1(I1, ... , Ii, ... , In)
. . . . . . . . . . . . . . .
G
k(I1, ... , Ii, ... , In)
. . . . . . . . . . . . . . .
G
r(I1, ... , Ii, ... , In)
   
P((J1, ... , Jj, ... , Jr)) = P1(J1, ... , Jj, ... , Jr)
. . . . . . . . . . . . . . .
P
s(J1, ... , Jj, ... , Jr)
. . . . . . . . . . . . . . .
P
m(J1, ... , Jj, ... , Jr)

Then the result of alignment P (by function align_) of r-dimensional array PatternDA in m-dimensional pattern Pattern space; and of next alignment G of n-dimensional array DA in r-dimensional array PatternDA space is alignment F of n-dimensional array DA in m-dimensional pattern Pattern space. F is a product (superposition) of P and G:

The coordinate alignment rules Fs (0 s m) from F mapping are a superposition of the coordinate alignment rules for P and G mappings:

1. Let Ps((J1, ... , Jr)) = { q Ms: 0 q MAXs } , where:

MAXs - maximum of the index variable of the s-th dimension of Pattern;
Ms - range of values of the index variable of the pattern Pattern s-th dimension.

Then Fs(I1, ... , In) ) = Ms, that is the full replication of the array DA along the s-th dimension of the pattern Pattern.

2. Let Ps((J1, ... , Jr)) = Cs*Jk + Ds and
Gk((I1, ... , In)) = Qk*Il + Rk ,

where:

Jk - index variable of the k-th dimension of the array PatternDA;
Il - index variable of the l-th dimension of the array DA;
Cs, Qs - integers;
Dk, Rk - nonnegative integers.

Then Fs((I1, ... , In)) = As*Il + Bs , where As = Cs*Qk and Bs = Cs*Rk + Ds.

3. Let Ps((J1, ... , Jr)) = Cs*Jk + Ds and
Gk((I1, ... , In)) = { q Mk: 0 q MAXk } ,

where:

Jk - index variable of the k-th dimension of the array PatternDA;
Cs - integer;
Ds - nonnegative integer;
MAXk - maximum of the index variable of the k-th dimension of the array PatternDA;
Mk - range of values of the index variable of the PatternDA array k-th dimension.

Then

that is the partial replication (stretched by the linear function Ps) of the array DA along the s-th dimension of the pattern Pattern.

The described alignment F is equivalent to the sequence of the alignments P and G. Consider the case, when the pattern Pattern is not an abstract machine representation. Then it is aligned by some function Q with another pattern. In this case, the alignment functions F and Q may be substituted by their superposition Q*F. So, the resulting alignment of the array DA with the abstract machine representation can be built substituting sequential alignment functions by their superposition (Run-Time System does it to align the array DA by the pattern PatternDA when the function align_ is called).

7.3 Realigning distributed array

long realn_( long
PatternRef
long
long
long
long
ArrayHeader[],
*PatternRefPtr,
AxisArray[],
CoeffArray[],
ConstArray[],
*NewSignPtr );
     
ArrayHeader - the header of the distributed array to be realigned.
*PatternRefPtr - reference to the alignment pattern.
AxisArray - array, which j-th element is a dimension number of the distributed array used in linear alignment rule for pattern (j+1)-th dimension.
CoeffArray - array, which j-th element is a coefficient for the distributed array index variable used in linear alignment rule for the pattern (j+1)-th dimension.
ConstArray - array, which j-th element is a constant used in the linear alignment rule for the pattern (j+1)-th dimension.
*NewSignPtr - the flag of updating of the distributed array (to be equal to 1).

The function realn_ cancels the allocation of the distributed array with header *ArrayHeaderPtr, defined previously by the function align_(or malign_, see section 7.5). The function sets a new allocation for this array according to the parameters AxisArray, CoefArray and ConstArray. If this array has been used as pattern of alignment of some arrays, nevertheless the allocations of these arrays are not changed.

As remapped distributed array, as alignment pattern must be mapped on the current processor system or its subsystem (perhaps, each one on its own).

The contents of redistributed arrays, depending on *NewSignPtr value, will be

If NewSignPtr = 1 redistributed array will be nevertheless cleared by "zero" byte, if Run-Time system was started with non-zero DisArrayFill parameter (sysdebug.* files of base parameter sets).

When the array is remapped, the values of its shadow edges (see section 12) are not kept.

The array can be realigned by function realn_ only, if it was created by the function crtda_ with parameter *ReDistrParPtr equal to 2 or 3 (see section 6).

The distributed array, specified in the function call, can be not mapped earlier. In this case realn_ function is executed as align_ function, but if NewSignPtr = 2 firstly distributed array will be cleared by "zero" byte. *ReDistrParPtr parameter value of the distributed array creation is irrelevant.

The function returns non-zero, if realigned array has a local part on the current processor, and zero in other case.

7.4 Requesting map

ArrayMapRef arrmap_( long
long
ArrayHeader[],
*StaticSignPtr );
     
ArrayHeader - header of the distributed array.
*StaticSignPtr - flag of static map creation.

The function arrmap_ creates an object (a map), describing current mapping of the distributed array onto the processor system representation, and returns a reference to the created object.

The map contains in particular the following information:

If flag of static map *StaticSignPtr is not equal to zero, then the map will not be deleted when control exits a program block (see section 8). Such map can be deleted only explicitly by function delarm_ considered in section 7.7.

7.5 Specifying distributed array mapping according to map

     
ArrayHeader - header of the distributed array.
*AMViewRefPtr - reference to the abstract machine representation, being alignment pattern.
*ArrayMapRefPtr - reference to map of the distributed array.

The function malign_ performs distributed array mapping onto abstract machine representation according to map.

The ranks of the distributed array and the abstract machine representation must be equal to corresponding ranks in map.

If AMViewRefPtr is equal to NULL or *AMViewRefPtr is equal to zero, then abstract machine representation, whose reference is in the map, will be used as alignment pattern.

Before malign_ function call the abstract machine representation (specified by *AMViewRefPtr reference or in the map) must be mapped on the current processor system or its subsystem.

It is not allowed to map already mapped distributed array.

The function returns non-zero, if mapped array has a local part on the current processor, and zero in the other case.

Note. If DisArrayFill parameter of Run-Time system startup is non-zero (sysdebug.* files of base parameter sets), mapped distributed array will be cleared by "zero" byte, i.e. by the byte with all zero bits). If DisArrayFill parameter is zero, the contents of mapped array is undefined.

7.6 Realigning distributed array according to map

long mrealn_( long
AMViewRef
ArrayMapRef
long
ArrayHeader[],
*AMViewRefPtr,
*ArrayMapRefPtr,
*NewSignPtr );
     
ArrayHeader - header of realigned distributed array.
*AMViewRefPtr - reference to abstract machine representation, being alignment pattern.
*ArrayMapRefPtr - reference to the map of distributed array.
*NewSignPtr - the flag of the distributed array updating (if it is equal to 1).

Function mrealn_ cancels allocation defined earlier by the function. malign_ (or align_ , see section 7.1) for the distributed array with header *ArrayHeaderPtr, and defines for the array new allocation according to specified map. The allocation of all distributed arrays, for whose alignment considered array was used as pattern, will not be changed.

The ranks of the distributed array and the abstract machine representation must be equal to corresponding ranks in the map.

If AMViewRefPtr is NULL or *AMViewRefPtr is zero, then abstract machine representation, whose reference is in the map, will be used as alignment pattern.

As remapped distributed array, as the abstract machine representation (specified by *AMViewRefPtr reference or in the map) must be mapped on the current processor subsystem or its subsystem (perhaps, each one on its own) when the function is called.

The contents of redistributed arrays, depending on *NewSignPtr value, will be

If NewSignPtr = 1 redistributed array will be nevertheless cleared by "zero" byte, if Run-Time system was started with non-zero DisArrayFill parameter (sysdebug.* files of base parameter sets).

When the array is remapped, the contents of its shadow edges (see section 12) are not kept.

The array can be realigned by function mrealn_ only in the case, if it was created by the function crtda_ with parameter *ReDistrParPtr equal to 2 or 3 (see section 6).

The distributed array, specified in the function call, can be not mapped earlier. In this case mrealn_ function is executed as malign_ function, but if NewSignPtr = 2 firstly distributed array will be cleared by "zero" byte. *ReDistrParPtr parameter value of the distributed array creation is irrelevant.

The function returns non-zero, if remapped array has a local part on the current processor, and zero in other case.

7.7 Deleting map

long delarm_ (ArrayMapRef *ArrayMapRefPtr);

*ArrayMapRefPtr - reference to the map of the distributed array.

The function deletes the map of the distributed array created by function arrmap_. After deleting the map the reference can be used by user program for its own goals.

The map can be deleted by delarm_ function only if it was created in the current subtask and in the current program block (or its sub-block) (see sections 8 and 10).

To delete map the function delobj_ can also be used (see section 17.5).

The function returns zero.

Note. One of the possible usages of the functions, aligning the distributed arrays according to the map is keeping and restoring the array allocation in procedures, dealing with external distributed arrays with their redistribution.

Note, that at the moment of using the map for mapping and remapping arrays, the array, being source for the map, may not exist.

7.8 Requesting reference to abstract machine representation, specified distributed array is mapped on

AMViewRef getamv_ (long ArrayHeader[]);

ArrayHeader - distributed array header.

The function getamv_ returns the reference to abstract machine representation, the distributed array, specified by ArrayHeader header, was aligned with. NULL value of returned reference means that given array is not mapped.


Lib-DVM interface description (contents) Part 1
(1-5)
Part 2
(6-7)
Part 3
(8-11)
Part 4
(12-13)
Part 5
(14-15)
Part 6
(16-18)
Part 7
(19)