Branching and joins

When two steps do not depend on each other, running them together cuts latency to the slower of the two rather than the sum. Fanning out is easy: give a node several outgoing edges and start them all.

The join is the hard part, and it is hard for reasons that have nothing to do with models.

• Waiting: the join cannot run until every branch has arrived, so one slow branch sets the pace.
• Failure: if one branch fails, does the join proceed with partial state, or does the whole thing fail? Both are defensible; the mistake is not deciding.
• Conflict: two branches writing the same key need a rule, decided in advance rather than by whichever finished last.
• Cost: parallel branches all spend tokens, including the ones whose results the join discards.

A useful pattern is speculative fan-out: start the expensive lookups you will probably need while a classifier is still deciding, then discard what turns out to be irrelevant. It trades tokens for latency, which is the right trade in an interactive product and the wrong one in a batch job.