Lib-DVM interface description (contents) | Part 1 (1-5) |
Part 2 (6-7) |
Part 3 (8-11) |
Part 4 (12-13) |
Part 5 (14-15) |
Part 6 (16-18) |
Part 7 (19) |
created: february, 2001 | - last edited 03.05.01 - |
6 Distributed array creating and deleting
6.1 Creating distributed array
long crtda_ ( | long long void long long long long long long long |
ArrayHeader[], *ExtHdrSignPtr, *BasePtr, *RankPtr, *TypeSizePtr, SizeArray[], *StaticSignPtr, *ReDistrParPtr, LowShdWidthArray[], HiShdWidthArray[] ); |
ArrayHeader | - | header of the array to be created. | |
*ExtHdrSignPtr | - | flag of extended header. | |
BasePtr | - | base pointer, used to access to the distributed array elements. | |
*RankPtr | - | rank of the created array. | |
*TypeSizePtr | - | size of the array element (in bytes). | |
SizeArray | - | array, which i-th element is a size of the (i+1)-th dimension of the created (0 £ i £ *RankPtr-1). | |
*StaticSignPtr | - | the flag of the static array creation. | |
*ReDistrParPtr | - | the permission flag of using of created distributed array as a parameter in the function redis_ and realn_ (see sections 5.2 and 7.3). | |
LowShdWidthArray | - | array, which i-th element is a width of the low shadow edge of the (i+1)-th dimension of the distributed array. | |
HiShdWidthArray | - | array, which i-th element is a width of the high shadow edge of the (i+1)-th dimension of the distributed array. |
The creation of a distributed array using function crtda_ means only an initialization of the internal system structures describing the array. The memory for the distributed array elements is allocated only after mapping this array on an abstract machine by the function align_. (In another words, memory is allocated after evaluating for each processor abstract machine it is mapped onto, a local part of distributed array according to its distribution, see section 7.1). An access to array elements is performed through Run-Time System functions (see section 13).
A header of the created array is an array of *RankPtr+1 (if *ExtHdrSignPtr = 0) or *RankPtr*2+2 (if *ExtHdrSignPtr = 1) elements of "long" type. The user program has to allocate this header in memory (whether statically or dynamically). Run-Time System only initializes this header. The main purpose of this header is to provide an effective access to the elements of the local part of the distributed array.
The base pointer BasePtr has to point to any variable of the same type as that of distributed array element. The value of this variable is not essential because the base pointer is only used to access to the elements from the local part of the distributed array.
The structure and usage of distributed array header (jointly with base pointer) to access to the array local part are considered in section 13.4.
If the flag *StaticSignPtr of the static array creation is not equal to zero, then the created distributed array is not deleted automatically when the control exits from the current program block (see section 8). Such array has to be deleted explicitly using the function delda_.
The parameter *ReDistrParPtr may have the following values:
0 | - | the array can not be an argument of the function redis_ and can not be redistributed by the function realn_; |
1 | - | the array can be an argument of the function redis_, but can not be redistributed by the function realn_; |
2 | - | the array can be redistributed by the function realn_, but this array can not be used in defining the abstract machine representation using the function redis_; |
3 | - | the array can be used as a reference to the abstract machine representation in the function redis_, and the array can be redistributed by the function realn_. |
The number of the parameters defined in the arrays LowShdWidthArray and HiShdWidthArray (that is the width of low and high shadow edges) has to be equal to the created array rank. See section 12 for more detail description of the shadow edges of the distributed array local parts. Note, that the shadow edge width can be equal to zero.
The function returns zero.
6.2 Deleting distributed array
long delda_(long ArrayHeader[]);
ArrayHeader – the header of the array to be deleted.
The function deletes the distributed array created by the function crtda_. After array deleting the user program can use distributed array header for any goal.
The distributed array can be deleted by delda_ function only if it was created in the current subtask and in the current program block (or its sub-block) (see sections 8 and 10). All edge groups, in which the array was included, (see section 12) will be deleted also.
A distributed array can be deleted also by the function delobj_ (see section 17.5).
The function returns zero.
6.3 Creating additional header of distributed array
long addhdr_ ( | long long |
NewArrayHeader[], ArrayHeader[] ); |
||
NewArrayHeader | - | header to be created. | ||
ArrayHeader | - | one of the existing headers. |
The function creates the distributed array header, which further will be used equitably with existing headers. In particular it can be used to create other additional headers.
When the distributed array is aligned by functions align_ and malign_ (see section 7) all its headers are initialized. The array realigning by the functions realn_, mrealn_, redis_ and mredis_ (see sections 5 and 7) also causes all its headers updating.
The function returns zero.
6.4 Deleting distributed array header
long delhdr_ (long ArrayHeader[]);
ArrayHeader - header to be deleted.
The function deletes the header, created by addhdr_ function or specified in crtda_ funcôion call. The only header of distributed array can't be deleted.
The function returns zero.
7.1 Aligning distributed array
long align_ ( | long PatternRef long long long |
ArrayHeader[], *PatternRefPtr, AxisArray[], CoeffArray[], ConstArray[] ); |
||
ArrayHeader | - | the header of the distributed array to be aligned. | ||
*PatternRefPtr | - | reference to the alignment pattern. | ||
AxisArray | - | array, which j-th element is a dimension number of the distributed array used in the linear alignment rule for the pattern (j+1)-th dimension. | ||
CoeffArray | - | array, which j-th element is a coefficient for distributed array index variable used in the linear alignment rule for the pattern (j+1)-th dimension. | ||
ConstArray | - | array, which j-th element is a constant used in the linear alignment rule for the pattern (j+1)-th dimension. |
The function align_ defines the location of the specified distributed array in the assigned abstract machine representation space (in other words the function aligns the distributed array). Besides, the elements of the array are distributed among the child abstract machines incorporating in the representation. The abstract machine representation is defined by means of the pattern, which can be as the representation itself, or a distributed array (this is called "indirect mapping"). In the last case the reference to the pattern is a first word of distributed array header.
Using the distributed array as a pattern is allowed only if this array has already been mapped on an abstract machine representation. Moreover, using an abstract machine representation as a pattern is allowed only if this representation has been mapped on a processor system. So the function align_ only distributes elements of the aligned distributed array among processors from this processor system.
Aligning pattern can be mapped (as final result) on the current processor system or its subsystem only.
It is not allowed to map already mapped distributed array.
Consider distributed array alignment in detail. Let F be a multifunctional with the domain of definition in a space of indexes of the aligned array and with the image in the space of indexes of the pattern:
F( (I_{1}, ... , I_{i}, ... , I_{n}) ) = | F_{1}(I_{1},
... , I_{i}, ... , I_{n})
´ . . . . . . . . . . . . .. . F_{j}(I_{1}, ... , I_{i}, ... , I_{n}) ´ . . . . . . . . . . . . .. . F_{m}(I_{1}, ... , I_{i}, ... , I_{n}) |
where:
´ | - | is a symbol of the Cartesian product; |
n | - | rank of the distributed array; |
m | - | rank of the pattern; |
I_{i} | - | index variable of the i-th dimension of the aligned array; |
F_{j} | - | multifunction, which range of value is a range of index variable of pattern j-th dimension. |
An alignment of the distributed array with the pattern is a defining the F multifunction. This defining designates that an element (I_{1}, ... , I_{n}) of an aligned array has to be located on the processor if and only if some pattern elements from F((I_{1}, ... , I_{n})) set are mapped on this processor. The set F((I_{1}, ... , I_{n})) is an image-set, and components of this set are vectors of space of indexes of the pattern.
The F_{1}, ... , F_{m} functions are named "coordinate alignment rules". Run-Time System provides the following set of alignment rules.
1. F_{j}(I_{1}, ... , I_{n}) = {A_{j}*I_{k }+ B_{j}} , where:
k=f(j) | - | dimension number of an aligned array (1 £ k £ n, f(j_{1}) ¹ f(j_{2}) when j_{1 }¹ j_{2}); |
A_{j} | - | integer; |
B_{j} | - | nonnegative integer. |
This alignment rule means that for each element (I_{1}, … , I_{n}) of index space of the aligned array the corresponding image-set contains one element – A_{j}*I_{k }+ B_{j}. This element belongs to the value range of the index variable of the pattern j-th dimension.
Note, that A_{j} and B_{j} have to meet the following requirements:
0 £ B_{j} £ MAX_{j}; and 0 £ A_{j}*MAX_{k} + B_{j} £ MAX_{j} , where:
MAX_{j} | - | maximum of the index variable of the j-th dimension; |
MAX_{k} | - | maximum of the index variable of the aligned array k-th dimension. |
2. F_{j}(I_{1}, ... , I_{n}) = {q Î M_{j}: 0 £ q £ MAX_{j}} , where:
M_{j} | - | range of values of the index value of the pattern j-th dimension; |
MAX_{j} | - | maximum of the index variable of the pattern j-th. |
This mapping rule means that for each element (I_{1}, … , I_{n}) of index space of the aligned array a corresponding image-set, consists of whole range of values of index variable of the pattern j-th dimension. In such a case, the symbol "*" ("any of the admissible") is usually used.
Examples.
Defining the aligned array distribution among pattern space (that is the defining F_{1}, ... , F_{m} functions) it is necessary to take into account that all elements of the distributed array have to be inside the pattern space. Observance of correct mapping on each stage of a chain of alignments guarantees the correct final distribution of array elements among the processors.
Note that if an array is used as a pattern, then after realignment of this array by the function realn_ (see section 7.3), the array aligned with this pattern keeps its location.
When the function align_ is called the parameters of the alignment rule F_{j}(I_{1}, ... , I_{n})= {A_{j}*I_{k }+ B_{j}} for the j-th pattern dimension have to be defined as follows:
AxisArray[j-1] | contains value k; |
CoeffArray[j-1] | contains value A_{j}; |
ConstArray[j-1] | contains value B_{j}. |
To define alignment rule F_{j}(I_{1}, ... , I_{n}) with image in set of all values of the index variable of j-th dimension of template for any I_{1}, ... , I_{n}, the value AxisArray[j-1] (k value) has to be set to -1. The values CoeffArray[j-1] and ConstArray[j-1] are irrelevant in this case.
The number of the alignment rules has to be equal to the rank of the pattern, when the function align_ is called.
The function returns non-zero value, if mapped array has a local part on the current processor, and zero otherwise.
Note 1. The distributed array mapping, specified for function align_, may be implemented only in the case, if for all the processors, at least one array element is mapped onto, a local size of each array dimension, obtained as a result of mapping, is not less than widths of high and low shadow edges of the dimension.
Note 2. If DisArrayFill parameter of Run-Time system startup is non-zero (sysdebug.* files of base parameter sets), mapped distributed array will be cleared by "zero" byte, i.e. by the byte with all zero bits). If DisArrayFill parameter is zero, the mapped array contents is undefined.
Let the function G defines the location of the n-dimensional array in the space of the r-dimensional pattern; and let the function P defines the location of the r-dimensional array in the space of the m-dimensional pattern:
G((I_{1}, ... , I_{i}, ... , I_{n})) = | G_{1}(I_{1},
... , I_{i}, ... , I_{n})
´ . . . . . . . . . . . . . . . G_{k}(I_{1}, ... , I_{i}, ... , I_{n}) ´ . . . . . . . . . . . . . . . G_{r}(I_{1}, ... , I_{i}, ... , I_{n}) |
P((J_{1}, ... , J_{j}, ... , J_{r})) = | P_{1}(J_{1},
... , J_{j},
... , J_{r}) ´ . . . . . . . . . . . . . . . P_{s}(J_{1}, ... , J_{j}, ... , J_{r}) ´ . . . . . . . . . . . . . . . P_{m}(J_{1}, ... , J_{j}, ... , J_{r}) |
Then the result of alignment P (by function align_) of r-dimensional array PatternDA in m-dimensional pattern Pattern space; and of next alignment G of n-dimensional array DA in r-dimensional array PatternDA space is alignment F of n-dimensional array DA in m-dimensional pattern Pattern space. F is a product (superposition) of P and G:
The coordinate alignment rules F_{s} (0 £ s £ m) from F mapping are a superposition of the coordinate alignment rules for P and G mappings:
1. Let P_{s}((J_{1}, ... , J_{r})) = { q Î M_{s}: 0 £ q £ MAX_{s}_{ }} , where:
MAX_{s} | - | maximum of the index variable of the s-th dimension of Pattern; |
M_{s} | - | range of values of the index variable of the pattern Pattern s-th dimension. |
Then F_{s}(I_{1}, ... , I_{n}) ) = M_{s}, that is the full replication of the array DA along the s-th dimension of the pattern Pattern.
2. | Let | P_{s}((J_{1},
... , J_{r})) = C_{s}*J_{k}
+ D_{s} and G_{k}((I_{1}, ... , I_{n})) = Q_{k}*I_{l} + R_{k} , |
where:
J_{k} | - | index variable of the k-th dimension of the array PatternDA; |
I_{l} | - | index variable of the l-th dimension of the array DA; |
C_{s}, Q_{s} | - | integers; |
D_{k}, R_{k} | - | nonnegative integers. |
Then F_{s}((I_{1}, ... , I_{n})) = A_{s}*I_{l} + B_{s} , where A_{s} = C_{s}*Q_{k} and B_{s} = C_{s}*R_{k} + D_{s}.
3. | Let | P_{s}((J_{1},
... , J_{r})) = C_{s}*J_{k}
+ D_{s} and G_{k}((I_{1}, ... , I_{n})) = { q Î M_{k}: 0 £ q £ MAX_{k }} , |
where:
J_{k} | - | index variable of the k-th dimension of the array PatternDA; |
C_{s} | - | integer; |
D_{s} | - | nonnegative integer; |
MAX_{k} | - | maximum of the index variable of the k-th dimension of the array PatternDA; |
M_{k} | - | range of values of the index variable of the PatternDA array k-th dimension. |
Then
that is the partial replication (stretched by the linear function P_{s}) of the array DA along the s-th dimension of the pattern Pattern.
The described alignment F is equivalent to the sequence of the alignments P and G. Consider the case, when the pattern Pattern is not an abstract machine representation. Then it is aligned by some function Q with another pattern. In this case, the alignment functions F and Q may be substituted by their superposition Q*F. So, the resulting alignment of the array DA with the abstract machine representation can be built substituting sequential alignment functions by their superposition (Run-Time System does it to align the array DA by the pattern PatternDA when the function align_ is called).
7.3 Realigning distributed array
long realn_( | long PatternRef long long long long |
ArrayHeader[], *PatternRefPtr, AxisArray[], CoeffArray[], ConstArray[], *NewSignPtr ); |
ArrayHeader | - | the header of the distributed array to be realigned. | ||
*PatternRefPtr | - | reference to the alignment pattern. | ||
AxisArray | - | array, which j-th element is a dimension number of the distributed array used in linear alignment rule for pattern (j+1)-th dimension. | ||
CoeffArray | - | array, which j-th element is a coefficient for the distributed array index variable used in linear alignment rule for the pattern (j+1)-th dimension. | ||
ConstArray | - | array, which j-th element is a constant used in the linear alignment rule for the pattern (j+1)-th dimension. | ||
*NewSignPtr | - | the flag of updating of the distributed array (to be equal to 1). |
The function realn_ cancels the allocation of the distributed array with header *ArrayHeaderPtr, defined previously by the function align_(or malign_, see section 7.5). The function sets a new allocation for this array according to the parameters AxisArray, CoefArray and ConstArray. If this array has been used as pattern of alignment of some arrays, nevertheless the allocations of these arrays are not changed.
As remapped distributed array, as alignment pattern must be mapped on the current processor system or its subsystem (perhaps, each one on its own).
The contents of redistributed arrays, depending on *NewSignPtr value, will be
If NewSignPtr = 1 redistributed array will be nevertheless cleared by "zero" byte, if Run-Time system was started with non-zero DisArrayFill parameter (sysdebug.* files of base parameter sets).
When the array is remapped, the values of its shadow edges (see section 12) are not kept.
The array can be realigned by function realn_ only, if it was created by the function crtda_ with parameter *ReDistrParPtr equal to 2 or 3 (see section 6).
The distributed array, specified in the function call, can be not mapped earlier. In this case realn_ function is executed as align_ function, but if NewSignPtr = 2 firstly distributed array will be cleared by "zero" byte. *ReDistrParPtr parameter value of the distributed array creation is irrelevant.
The function returns non-zero, if realigned array has a local part on the current processor, and zero in other case.
ArrayMapRef arrmap_( | long long |
ArrayHeader[], *StaticSignPtr ); |
ArrayHeader | - | header of the distributed array. | ||
*StaticSignPtr | - | flag of static map creation. |
The function arrmap_ creates an object (a map), describing current mapping of the distributed array onto the processor system representation, and returns a reference to the created object.
The map contains in particular the following information:
If flag of static map *StaticSignPtr is not equal to zero, then the map will not be deleted when control exits a program block (see section 8). Such map can be deleted only explicitly by function delarm_ considered in section 7.7.
7.5 Specifying distributed array mapping according to map
ArrayHeader | - | header of the distributed array. | ||
*AMViewRefPtr | - | reference to the abstract machine representation, being alignment pattern. | ||
*ArrayMapRefPtr | - | reference to map of the distributed array. |
The function malign_ performs distributed array mapping onto abstract machine representation according to map.
The ranks of the distributed array and the abstract machine representation must be equal to corresponding ranks in map.
If AMViewRefPtr is equal to NULL or *AMViewRefPtr is equal to zero, then abstract machine representation, whose reference is in the map, will be used as alignment pattern.
Before malign_ function call the abstract machine representation (specified by *AMViewRefPtr reference or in the map) must be mapped on the current processor system or its subsystem.
It is not allowed to map already mapped distributed array.
The function returns non-zero, if mapped array has a local part on the current processor, and zero in the other case.
Note. If DisArrayFill parameter of Run-Time system startup is non-zero (sysdebug.* files of base parameter sets), mapped distributed array will be cleared by "zero" byte, i.e. by the byte with all zero bits). If DisArrayFill parameter is zero, the contents of mapped array is undefined.
7.6 Realigning distributed array according to map
long mrealn_( | long AMViewRef ArrayMapRef long |
ArrayHeader[], *AMViewRefPtr, *ArrayMapRefPtr, *NewSignPtr ); |
ArrayHeader | - | header of realigned distributed array. | ||
*AMViewRefPtr | - | reference to abstract machine representation, being alignment pattern. | ||
*ArrayMapRefPtr | - | reference to the map of distributed array. | ||
*NewSignPtr | - | the flag of the distributed array updating (if it is equal to 1). |
Function mrealn_ cancels allocation defined earlier by the function. malign_ (or align_ , see section 7.1) for the distributed array with header *ArrayHeaderPtr, and defines for the array new allocation according to specified map. The allocation of all distributed arrays, for whose alignment considered array was used as pattern, will not be changed.
The ranks of the distributed array and the abstract machine representation must be equal to corresponding ranks in the map.
If AMViewRefPtr is NULL or *AMViewRefPtr is zero, then abstract machine representation, whose reference is in the map, will be used as alignment pattern.
As remapped distributed array, as the abstract machine representation (specified by *AMViewRefPtr reference or in the map) must be mapped on the current processor subsystem or its subsystem (perhaps, each one on its own) when the function is called.
The contents of redistributed arrays, depending on *NewSignPtr value, will be
If NewSignPtr = 1 redistributed array will be nevertheless cleared by "zero" byte, if Run-Time system was started with non-zero DisArrayFill parameter (sysdebug.* files of base parameter sets).
When the array is remapped, the contents of its shadow edges (see section 12) are not kept.
The array can be realigned by function mrealn_ only in the case, if it was created by the function crtda_ with parameter *ReDistrParPtr equal to 2 or 3 (see section 6).
The distributed array, specified in the function call, can be not mapped earlier. In this case mrealn_ function is executed as malign_ function, but if NewSignPtr = 2 firstly distributed array will be cleared by "zero" byte. *ReDistrParPtr parameter value of the distributed array creation is irrelevant.
The function returns non-zero, if remapped array has a local part on the current processor, and zero in other case.
long delarm_ (ArrayMapRef *ArrayMapRefPtr);
*ArrayMapRefPtr - reference to the map of the distributed array.
The function deletes the map of the distributed array created by function arrmap_. After deleting the map the reference can be used by user program for its own goals.
The map can be deleted by delarm_ function only if it was created in the current subtask and in the current program block (or its sub-block) (see sections 8 and 10).
To delete map the function delobj_ can also be used (see section 17.5).
The function returns zero.
Note. One of the possible usages of the functions, aligning the distributed arrays according to the map is keeping and restoring the array allocation in procedures, dealing with external distributed arrays with their redistribution.
Note, that at the moment of using the map for mapping and remapping arrays, the array, being source for the map, may not exist.
AMViewRef getamv_ (long ArrayHeader[]);
ArrayHeader - distributed array header.
The function getamv_ returns the reference to abstract machine representation, the distributed array, specified by ArrayHeader header, was aligned with. NULL value of returned reference means that given array is not mapped.