A phase requires all the nodes in the phase's spatial extent to run coordinated router schedules. Once a given node has completed the communication and computation relevant to that phase, it instructs the router to shift to the next phase. Once some nodes have shifted to the next phase, data communication can continue at will for the new phase. Only when all the nodes are in the new phase can the compiler guarantee that, for example, a long-distance stream communication between two nodes will be able to deliver data from one processor to the other.
An important restriction is that nodes must always switch to the correct next phase. For example, if a given phase is optional, it must be skipped or not skipped uniformly by all the nodes involved. Similarly, if a node is involved in some phases and not in others, it must be careful to sequence through the correct phases; if a node is in some phase's spatial extent, it must be careful to run the schedule for that phase at the appropriate time.
Before a node can switch to the next phase, all communications that involve the nodes must have finished. (Communications from continuing operators are exempt from this requirement, as their communications will continue to be supported in the following phase.) Simply waiting until the last I/O word has been written to the last operator in the phase is not sufficient. All written data may not have left the router (or indeed left the processor interface); additionally, data from other nodes may still be in transit through the node in question. In either of these cases, the processor can't instruct the router to change phases, or the data will be stranded in the previous phase (or possibly injected inappropriately into a new operator in a subsequent phase, depending on whether the VFSM in question is reused later).
The text below assumes that phase changes are always issued by the processor. In fact, a phase change could be issued by neighbor nodes as well. One simple way to handle phase changing would be to have a final barrier (e.g., a reduction across the phase), followed by a multicast that directly instructed the routers to change phase. Doing so is not straightforward with a simple scheduled router, however. As discussed in Section 5.4.1, there may be different schedules resident in different routers. While the correct schedule could be guaranteed to be available somewhere in the router, special handling would be needed to ensure that the correct schedule was started on all the involved nodes. Accordingly, the request to change phases is simply passed through the processor on each node.
There are three issues involved in terminating a phase. The first is terminating a single operator in isolation; the second is figuring out which operators are relevant for phase termination; and the last is using this information to determine how to terminate the phase.