Attendance:

Ken Cain
Dennis Cottel
Rossen Dimitrov 
Jon Greene
Randy Judd
Jamie Kenny
Steve Paavola
Myra Jean Prelle
Brian Sroka

 

Agenda

1. Buffer sharing and I/O overlap issue: the num_buffers argument to
DRI_Reorg_create_*

from the 10/31/2001 working meeting: "consider R1-S and R1-R in a clique
reorg, each created with num_buffers=1. Does this mean that both sides
of reorg can have a buffer checked out (via Reorg_get_datapart) at the
same time, or only 1 side can use a single buffer at a time?"


2. Discuss API mechanisms for specifying buffer sharing

2a. review what was discussed at the 10/31/2001 working meeting, and
August 2001 general meeting
2b. define what use cases DRI should support
2c. assemble the suggested approaches, target use cases, attempt
resolution


3. Discuss DRI commit interface approaches

3a. identify the fundamental phases that must be executed before
DRI_Reorgs can be finalized

3b. evaluate current network-wide commit vs. a fine-grained per-Reorg
connect vs. hybrid approaches (e.g., deferred commit evaluation). Map
any discussed interfaces onto the fundamental commit phases identified
in agenda item 3a.

Agenda item #1: Buffer sharing and I/O overlap issue

Discussion 1-A:

Consider the following case:

• clique data reorg design
• both sides provide num_buffers = 1 to Reorg_create 
• question: is there 1 real buffer, and only 1 of the 2 DRI_Reorg objects can have it checked out at any time? Alternatively, are there 2 buffers, enough for each reorg to have a single buffer checked out?

steve: we don't have enough information about what the application will do, so the safest thing is to actually allocate 2 buffers, and enforce a rule that num_buffers refers to how many buffers affiliated with the reorg that can be checked out at any one time

dennis: concurs

steve: user could use the "options" flag to Reorg_create to disambiguate the 2 choices

ken: we also talked about bufferset object being used as a tool to disambiguate in this case

myra: suggests that the interpretation of num_buffers should be the degree of multi-buffering (2=double, 3=triple, ...)

Resolutions for discussion 1-A:

group: agrees (without a straw vote) that in the above described case, 2 buffers should be created, one for each DRI_Reorg  object

Discussion 1-B:

myra: would like to know when I say num_buffers=n, exactly how much user exposed and temporary/private memory will be created.
This is best left as a documentation task for the implementor. It doesn't have to be in the spec.

rossen: why not specify a run-time query function to expose what the implementation is doing with buffer allocation?

Resolutions for discussion 1-B:

group concurs with myra (without straw vote) that implementor's documentation can serve this purpose.

Discussion 1-C:

steve: suggests deleting Reorg_create_user from spec. The portable functionality for DRI would be defined as DRI_Reorg_create_system is today. Vendors engcouraged to do their own implementation-specific approach for buffer sharing and user-allocated memory. This way we get some collective experience with buffer sharing before codifying it in the spec.

this discussion was continued in agenda item #2

Resolutions for discussion 1-C (see discussion 2-B)

Agenda item #2: buffer sharing

Discussion 2-A: general discussion of buffer sharing and user-allocated memory

steve: concerned that if we specify user allocated memory and buffer sharing in 1.0, then all implementations have to provide it (even if those implementations do not recommend using those features) 

ken: user allocated memory is important to have in the 1.0 spec (in keeping w/ current programming practice)
ken: buffer sharing is important to have in 1.0 spec (for same reasons) 

group: considers system memory case allowing implementation to have flexibility, and user memory case is where you can specify sharing 

dennis: agrees that there needs to be a case where the user can bind user memory to a Reorg (but not necessarily in favor of re-introducing bufferset object to do this)

Resolutions for discussion 2-A (see discussion 2-B)

Discussion 2-B: review August 2001 meeting minutes, candidate buffer sharing approach #1 

For reference, here is what was suggested in August (pasted from meeting minutes):

• summary: using per-Reorg connect calls, and DRI_Bufferset objects:
• for system-alllocated memory:
• DRI_Bufferset_create (num_buffers, &bs_handle)
• DRI_Reorg_create(network, side, name, dataspec, group, dist, bs_handle, options, &reorg1) /* internal state of bs_handle is modified to reflect data sizes associated with reorg1 */
• DRI_Reorg_create(network, side, name, dataspec, group, dist, bs_handle, options, &reorg2) /* internal state of bs_handle is modified to reflect worst case data size among reorg1 and reorg2 */
• DRI_Reorg_connect(reorg1); /* reorg1 notices that the memory for its bufferset, bs_handle, has not yet been allocated. So, memory allocation is performed at this time */
• DRI_Reorg_connect(reorg2); /* reorg2 connect notices that the memory for its bufferset, bs_handle, has been allocated already in a prior connect call */
• <no longer need DRI_Reorg_process_inplace to express in-place processing buffer sharing relationships>
• In above examples, we need to consider what happens when user wants to express NO sharing between the reorgs created
• dennis: suggests a different version of DRI_Reorg_create that would tell the system to create a distinct bufferset internally
• application would have to pass num_buffers input argument instead of a bufferset handle
• benefit: user does not have to go through cumbersome approach of constructing a DRI_Bufferset object for each DRI_Reorg object
• For user-allocated buffers:
• DRI_Bufferset_create (num_buffers, &bs_handle)
• DRI_Reorg_create(network, side, name, dataspec, group, dist, bs_handle, options, &reorg1)
• DRI_Reorg_create(network, side, name, dataspec, group, dist, bs_handle, options, &reorg2)
• DRI_Bufferset_get_size (bs_handle, &nbytes)
• <user allocates num_buffers regions of memory, each of size nbytes)
• DRI_Bufferset_bind(bs_handle, void *ptrs[num_buffers]);
• DRI_Reorg_connect(reorg1); /* this function notices that bs_handle [referenced by reorg1] has user memory bound to it already, and so there is no memory allocation to be done */
• DRI_Reorg_connect(reorg2);/* this function notices that bs_handle [referenced by reorg2]  has user memory bound to it already, and so there is no memory allocation to be done */
• <no longer need DRI_Reorg_process_inplace to express in-place processing buffer sharing relationships>
• In above examples, we need to consider clique connections

steve: problematic case is if you give the same bufferset object B to an upstream reorg, and a downstream reorg (the processing takes place between them). If you do this with num_buffers = 1, then doesn't that imply in-place processing has to occur?
group: yes it would seem to be so (it implies the same as if DRI_Reorg_process_inplace was called with the upstream and downstream reorgs as arguments)

myra: questioned why the bufferset object has been removed from the spec, what was the rationale behind the decision?

dennis: suggests that this and other ambiguous cases were the reasons why the bufferset object was decided against at one point

group: considers removing user-allocated memory and buffer sharing from the spec. 

jon/myra: what about a placeholder object that would be passed into Reorg_create that would be the place where non-standard memory specifications could be made? 

dennis: only in favor of supplying virtual memory address pointers as arguments, not an object

myra: virtual addresses may not be extensible enough; consider using a DRI-like interface to a streaming I/O device, for which you may only have a physical address

group: agrees to forego user-allocated memory in conjunction with DRI 1.0. Implementors are encouraged to provide such features with non-portable interfaces, and this will define the list of candidate approaches that could be specified in a subsequent DRI specification.

Resolutions for discussion 2-B:

eliminate callback function approach to binding user memory i n DRI_Reorg_create_user: nobody is happy with it in this form
DRI_Reorg_create_system is renamed DRI_Reorg_create, functionality remains the same
No portable interfaces for user-allocated memory
No portable interfaces for buffer sharing, with the exception of DRI_Reorg_process_inplace

STRAW VOTE on all of the above points: 5 for /1 against / 3 abstain

 

Agenda Item #3: "connect" (Reorg finalization) discussion

in earlier discussions group agreed that DRI_connect is scoped to only the processes with whom the caller interacts (as opposed to a barrier synchronization among all processes in the scope of the DRI_Network object).

some question as to how to shut down a reorg
steve: should call DRI_disconnect(), then Reorg_disconnect
myra: problem with this is that Disconnect takes the network parameter, and that may involve a long list of Reorgs, not just the one you want to disconnect

dennis: we should re-investigate the idea of per-Reorg connect/disconnect granularity; we decided during the August 2001 meeting that this would  be a good thing to do.

rossen: is there any reason to disconnect a Reorg, and then re-use the same reorg object by reconnecting it?

ken/jon: now that we have decided that buffer sharing is not part of the standard, it really would make sense to be able to destroy  an existing set of DRI_Reorgs, and then re-create new ones corresponding to a new mode. If you pre-plan a bunch of modes by building all of the DRI_Reorg objects, then they may be forced to use separate memory.

dennis: how about DRI_Reorg_create, DRI_Reorg_connect, DRI_Reorg_disconnect, DRI_Reorg_destroy? User would pick whether to use the individual Reorg connect approach or the library-scoped DRI_connect/disconnect approach.  

dri_reorg_create

dri_reorg_connect ... dri_reorg_disconnect   -- OR --
dri_connect ... dri_disconnect

dri_reorg_destroy

The two approaches could not be mixed in the same program. 

For connect/disconnect at the Reorg object granularity, all create calls would need to be performed before the first DRI_Reorg_connect

dri_connect, dri_disconnect, dri_reorg_disconnect are all blocking

dri_reorg_disconnect reverts the state of the reorg back to what it was after create, but before reorg_connect
 

dennis: wants both individual and global connect/disconnects. individual reorg connect/disconnect is necessary politically (coarse-grained connect has been not well accepted part of MPI/RT spec). Use-case within one program must be symmetric; you pick one approach or the other as a user.
 

jon: proposal

4 functions:
 - 2 at reorg granularity
 - 2 at library wide granularity
 - none of the calls are barriers
 - none of the calls are synchronous
 - all of the connect calls block waiting for the corresponding reorg creates to have finished on the other side
 - all of the calls support a flag to control blocking or non-blocking behavior
 - the disconnect issue is not yet fleshed out in this proposal (see myra concern below)

myra: wants more time to figure out whether we can do a disconnect without a barrier

Resolutions for agenda item 3

group agrees (without straw vote)  to provide the two API options suggested by dennis. Details to be worked out following this meeting

Miscellaneous:

steve: we need to improve DRI_Reorg_create documentation (description of  DRI_REORG_STATIC_ONLY flag parameter value)

myra/jamie: does dri_put_datapart ever block? like when you can't actually write to the destination. Can the implementation wait until a dma engine can be fired, and then return? Or ,must it queue the buffer immediately, and then return to the user (which would require a progress engine).