Implement reduction-based `ParallelDim`

[lockshaw]

This refactors ParallelDim from the Unity replica-dim version to the improved model based on specific operators that relate to the non-parallel semantics

[wmdi]

There are two issues to fix:

1. In the old FF ParallelDim, we cannot distinguish between shards that are duplicated and shards that are partial sums. In the following example, we both get a parallel tensor with shape ((size=_, deg=2), (size=a, deg=1), (size=a, deg=1)). Recall that in the old FF codebase, each ParallelTensor has a ParallelTensorShape, which represents the parallel shape of the tensor through a list of ParallelDim's, and specially, the dimension indexed by 0 is the replica dimension. Here, when we apply the reduce operator to the results, we do not know whether to sum them up.

Our solution is to extend the parallel dimension to have the relationship between shards. Specifically, we will add one more attribute whose value should be one of sum'(+), equal(=), split(||) or not important`(). In this way, the result tensor shapes in the example above becomes ((size=, deg=2, type='='), (size=a, deg=1, type=''), (size=a, deg=1, type='')) and ((size=, deg=2, type='+'), (size=a, deg=1, type=''), (size=a, deg=1, type='_')), and then we know what to do to reduce the results.

Note that for some operators, the tensors involved in the computation are not fully represented in the PCG. Take Linear as an example. The core computation of Linear is a matrix multiplication, but the weight tensor is not in the PCG, which means we cannot derive its tensor shape from previous operators. However, we believe that there is a unique tensor shape for it to make the computation valid.

2. The runtime does not know the mapping between shards and machines. In the following example where we are doing addition with broadcast, we should make sure 1 and 5, 4 and 7, 2 and 6, 3 and 8 are in the same machine. We address this by maintaining a mapping of dimensions that should match (the top part of the figure). This will provide enough information for the runtime to store tensor shards.

The plan for implementing this is:

1. Refactor ParallelDim to support the extended semantic;
2. Refactor get_output_shape for parallel tensors: we do both tensor shape inference and dimension mapping here, since they share a lot of information;
3. Refactor the compiler part to maintain the dimension mapping for each operator.

After this, the whole pipeline of compiler is:

1. Read the user-defined model;
2. Apply tensor shape inference and get the computation graph;
3. View it as a parallel computation graph as the start point of optimization;
4. Apply Unity search, which involves applying substitutions to the PCG, and getting a PCG without parallel tensor shapes or dimension mappings;
5. Apply parallel tensor shape inference and dimension mapping inference and get the full PCG.

[lockshaw]

Closed by #1394