Data Reorg meeting minutes, 6/8/2000 and 6/9/2000

• Approach 1: Implement a subset of the API (some functions are missing)
• Approach 2: Implement the API, but restrict functionality (all key functions present, but not all data reorgs supported)
• To date, approach 2 has been the prevailing thinking of the group
• Group agrees that approach 2 will continue to be used in API specification activities. Limitations in functionality by different implementations will be an area of competition
• Group would be in favor of specifying levels of acceptable functionality at some point

• Source code level: same source produces same final result (but perhaps the default partitioning at intermediate stages is slightly different)
• Fine grained semantic level: same source produces same assignment of data to processors,  in addition to computing same result
• This is still an open issue

Myra: proposing in LIS API that we might want to supply lots of detailed (previously specified) info to dri_bufferset_create. In fact, you may even need per-process detailed partitioning information (if you are layering on something like Mercury's PAS, for example). Options:

• Have bufferset_create do the right thing (local scope allocation only), and make no API changes
• Additionally, specify (again, within local scope) data alignment of the memory that gets allocated for a buffer
• Pass in additional information (datatype, local memory layout, data distribution object, ...) that could help create the local buffers
• Finally,  specify per-process information as part of the bufferset_create argument list (although this is motivated by PAS distributed buffers)

A big problem is looming regarding data alignment between communication and computation library APIs. Creating buffers in DRI, then trying to compute in VSIPL. You could admit the DRI created buffers to VSIPL, but if the data being operated on by VSIPL isn't sufficiently aligned, memory copies could happen under the hood in VSIPL impl.

The general area of layouts and memory allocation needs to be revisited

Strong desire by group to allow users to admit their own buffers to Data Reorg
 

API change proposals:

dri_bufferset_create (IN nbufs, IN dist, OUT bufset)
 - if nbufs is 0 creates an empty bufferset

dri_bufferset_add_buffer(IN bufaddress, INOUT bufset)
 - user would call this one time per buffer that needs to be included in the bufferset

Alternative:
dri_bufferset_import (IN nbufs, IN [ ] buffers, OUT bufset)
 [ this avoids the user doing a dri_bufferset_add_buffer on a previously created bufferset from dri_bufferset_create ]
 

Group agrees that the bufferset_import approach is probably more useful, and avoids overloading the intent of dri_bufferset_create (which performs implementation-provided memory allocation services)
 
 

Jon: proposes a new calc_size routine (simpler name, and returns more than just size - maybe an opaque object)
Ken: minimal interface should allow query for buffer size (already present), and then the beginning alignment (needs introduction into LIS API)
The BIG question here is "are these 2 integer quantities sufficient"?
 

Co-layer and  related issues discussion

What are the possibilities?

• LIS API that contains all necessary support for all aspects of Data Reorg (including "peripheral" areas such as process sets, data types, ...). Risk of re-inventing the message passing wheel in some cases. Could lead to applications that use only DR.
• LIS API w/ minimal specification of peripheral support services, relying on a "co-layer" approach (use MPI communicators, data types, MPI/RT groups, datatypes, etc.).  Optionally, re-use core communication constructs from other middleware (e.g.,  MPI/RT channels). Could lead to applications that use mostly DR for the hard data redistribution problems, and MPI or MPI/RT for control/parameter information exchange.
• A "late-binding" Data Reorg co-layer with MPI. Preserves send/recv function naming of MPI (but level of abstraction is at communicator level now). Also allows user allocated memory  buffers to be specified at reorg-time (and not in advance as is the case with an early-binding form of DR).
• <many others> ...

Decision: The group should acknowledge the wide spectrum of possible implementations of DR. Perhaps write up conceived instantiations in the final report. The question is: which MPI co-layer form of DR are we going to produce in the final "product" of this activity (we have discussed producing such a specification as part of our work)? A classical "co-layer" approach that preserves early-binding focus,  or one that preserves an MPI-like (late-binding) set of interfaces?

Resolution: The LIS API will consist of 2 categories:

• Minimal functionality, "standalone" interfaces (that perform similar functions that are already found in MPI, MPI/RT, etc.).  Applications may or may not end up using these interfaces, based on the particulars of the DR implementation used. Most opportunity to forego the use of these routines will be in the areas of data types and process sets. Advanced co-layer implementations may allow channel operations to be merged (e.g., with MPI/RT).
• "CORE" interfaces that are inherently unique to DR.

List of "minimal, standalone" functions:
 
 

Function	Related Object(s)
DRI_Init	DRI_Group (DRI_GROUP_WORLD pre-defined group - see below)
DRI_Finalize
DRI_Group_create	DRI_Group
DRI_Group_get_rank	DRI_Group
DRI_Group_get_size	DRI_Group
DRI_Bufferset_system_create	DRI_Bufferset, DRI_Buffer_Id
DRI_Bufferset_user_create	DRI_Bufferset, DRI_Buffer_Id
DRI_Channel_create_send	DRI_Channel
DRI_Channel_create_recv	DRI_Channel
DRI_Channel_connect	DRI_Channel
DRI_Channel_connect_sendrecv	DRI_Channel
DRI_Channel_get	DRI_Channel
DRI_Channel_put	DRI_Channel

 

List of "DRI Core" functions - those that MUST be provided everywhere, regardless of underlying implementation approach:
 
 
 

Function	Related Object(s)
DRI_Global_data_create	DRI_Global_data
DRI_Overlap_create	DRI_Overlap
DRI_Partition_block_create	DRI_Partition
DRI_Partition_blockcyclic_create	DRI_Partition
DRI_Partition_whole_create	DRI_Partition
DRI_Distribution_create	DRI_Distribution
DRI_Layout_create (or comparable equivalent when specified)	DRI_Layout

 
 
 

Focus on process sets:

Group agrees to provide a DRI_GROUP_WORLD predefined object, and create sub-groups that are subsets of DRI_GROUP_WORLD.
DRI_Init implementation could construct DRI_GROUP_WORLD as a side-effect.

DRI_group_create (IN original_group, IN list_of_ranks, IN num_ranks, OUT new_group)
 

Miscellaneous changes to some data types in the LIS API:

DECISION: DRI_distspec type is now renamed to DRI_Partition

DECISION: DRI_dist is now renamed to DRI_Distribution

**** CORRECTION needed: dri_global_data_create is listed twice: once for the create function in earnest, one for the destroy function (didn't copy and paste and edit properly)
 

Specific API issues discussed
 

Talking about put/get channel interface and interaction with buffersets. The issue of setting up efficient dma transfers in advance was raised. Three areas of concern were discussed:

• What happens when the number of buffers associated with the send-side of a channel differs from the number of buffers on the receive-side? How will the middleware ensure optimized dma transfer performance (usually requiring precise in-advance knowledge of which buffers are involved in a particular transfer)?
• What happens when the number of buffers on one side of a channel differ from process to process (within the same process group that is on the same "side" of the data reorganization)?
• What happens when application codes call the put/get operations in an unanticipated/unintended order? (e.g., put/put/get instead of put/get/put/get/...)

Group agrees that the following restrictions should be imposed to facilitate high performance

• on the send-side of a channel, all participating send processes must have the same number of buffers in their buffer sets associated with that channel
• same applies for the receive-side of a channel

Dennis/Arkady: need somethiing similar to buffer iterator "policies" at bufferset creation time. This basically enforces an ordering on subsequent operations.

This topic area may need some additional thought offline.
 

Buffersets discussion:

Do we need buffersets? Can they just be an internal object to channels? Would perhaps simplify the user api? (but then if you want to share the internal bufferset among channels (e.g., on the same process, use the same bufferset for received "upstream" data, and data to be sent "downstream" to another set of processes following some processing) then we would need some type of channel "clone" operation)
 

What are our products?
 

• "CORE" interfaces (see tables above)
• Standalone API (see tables above)
• An MPI-ish instantiation of the data reorg concept
• Final document

What's our near-term plan

• 2 meetings prior to HPEC 2000 (August and September in Boston area)
• Approx 4 total meetings needed to complete this group's activity
• resolve remaining core and stand-alone functionality designations (including bufferset/buffer issues)
• fully explore impact of Mercury-submitted layout ideas on the "core" objects and functions. Are there even more flexible approaches that could change this critical path of objects and functions
• MPI-ish instantiation of Data Reorg
• Final document

Mercury proposal regarding memory layout objects, and a new proposed "memory descriptor object" (MDO)

THE DETAILS OF THIS ARE YET TO BE OFFICIALLY PUT IN THE MINUTES. It requires some post-meeting coordination with the principals to verify  that what was recorded is accurate.

Details will be published under separate cover, and presumably posted on www.data-re.org in the "Meetings" section, along with the official minutes.