Random transforms for both input and target?

[fmassa]

In some scenarios (like semantic segmentation), we might want to apply the same random transform to both the input and the GT labels (cropping, flip, rotation, etc).
I think we can get this behaviour emulated in a segmentation dataset class by resetting the random seed before calling the transform for the labels.
This sound a bit fragile though.

One other possibility is to have the transforms accept both inputs and targets as arguments.

Do you have any better solutions?

[soumith]

hmm, good point.
We could make a stateful transform object that resets the random seed every 2 steps.
To do this you simply have to create a class with the call operator that is stateful.

For example:

trans = transforms.tworandom(transforms.compose([...]))
dataset = dset.SegmentationDset(...,transform=trans, target_transform=trans)

[colesbury]

Ewww, that's a fragile ugly hack. I also don't see how it will help, since the target isn't an image.

Can't you just extend the CocoDataset? Generate the random parameters and then apply the same logical op to both image and target:

class TransformedCocoDataset(CocoDataset):
  def __getitem__(self, i):
    input, target = super(TransformedCocoDataset, self).__getitem__(i)
    hflip = random.random() < 0.5
    if hflip:
      image.hflip(image)
      # flip target rectangle
    # etc.
    return input, target

Perhaps we should provide similar operations that work on bounding boxes?

[colesbury]

Oh -- I misread the issue. I guess the target is an image for segmentation. It still seems to me like you want to generate the random parameters once and apply the operation twice. For things that aren't trivial (like RandomSizedCrop), we may want to refactor out the part that generates the random parameters to the image op. Probably don't have to do anything for trivial ops like horizontal-flip.

[fmassa]

I think the solution proposed by @colesbury about sub-classing on the dataset is the most general one.
In a maybe cleaner way, this solution is actually equivalent to using a transformdataset from tnt with a single callable instead of a dict of callables.

Also, the current way of passing transform and target_transform in every dataset is equivalent to using a transformdataset with dicts of transforms as input (and returning dicts as well instead of tuples).

As such, are you ok if we merge tnt datasets into core, and remove transform and target_transform arguments from vision datasets?

[ClementPinard]

The way I see it in @colesbury code, we will have the same probleme when trying to compose different transform functions, because random parameters are created within the call function. we won't be able to customize transform functions, and will have to create a subdataset per set of transform functions we want to try.

What about special transformation for both imputs and targets ? This may create some duplicates functions like randomcrop for image based target (maybe add target as an optional argument ?) , but i don't see how we would apply properly coherent transformations for input and target.

We could also give a seed as an argument in addition to img in getitem , but we are not garanteed transform and target_transform will be constructed in a dual manner such that random variables will have coherent effects (like flipping both inputs and targets, or translating BB coordinates according to input img transformation)

I feel like there should 3 types of transform : transform_input that deals with transformations that are independent of target, like flip-crop for classification, transform_target idem for target and lastly co_transform(sorry about bad terminology) that deals with dependent transformations and must take input and target as arguments and I believe concerns the vast majority of data augmentation for more geometric problems, such as detection, depth estimation, flow, etc

[abeschneider]

I actually just ran into this problem myself. Another potential solution is to allow transforms to perform the same operation over a list of images. Thus, CocoDataset would look like:

if self.transform is not None:
    img, label = self.transform(img, label)

and the transform itself may look like:

    def __call__(self, *images):
        # perform some check to make sure images are all the same size
        if self.padding > 0:
            images = [ImageOps.expand(img, border=self.padding, fill=0) for im in images]

        w, h = images[0].size
        th, tw = self.size
        if w == tw and h == th:
            return images

        x1 = random.randint(0, w - tw)
        y1 = random.randint(0, h - th)
        return [img.crop((x1, y1, x1 + tw, y1 + th)) for img in images]

It may be possible to further abstract this out and create a container class for an image that automatically applies the same operations across the collection. That way the operations could be agnostic to what they are actually operating on.

[fmassa]

I gave it a shot on implementing some generic random transform that could be applied so several inputs (even for example images + bounding boxes). An example implementation can be found here.
Here are the main ideas:

• Allow to pass an arbitrarily nested tuple of inputs (can be image, labels, bboxes, masks, etc), and the Compose transform can accept the tuple instead of individual elements
• The sequence of transforms should be a list as before, but if an entry is a list, then it is applied to the corresponding instance of the tuple of inputs, and if it's not, then it's applied to all the elements at the same time.
• Every random transform should consist of 2 classes: a random generator, that generates the elements during the function call, and the transform itself, that can accept the generator object and uses the random parameters generated from it. It can thus allow the same transformation to be applied to different inputs by reusing the same implementation, and it's not needed to implement say RandomCrop2Images etc.
• It's backward compatible with previous transforms.

It's a bit rough (and maybe a bit too complicated), but it could handle the cases we have mentioned in this thread.

[bodokaiser]

@fmassa does your proposal also consider transforms which depend on input and target image?

This would be helpful when you have for example want to crop your input image to the size of the target image.

[fmassa]

@bodokaiser Yes, this should be handled as well.
All you need to do is to create a transform that take both input and target and perform the operation, returning input and target. Here is an example:

class MyJointOp(object):
    def __call__(self, input, target):
        # perform something on input and target
        return input, target

and then you would use it as follows (using my transforms), supposing that your dataset outputs input and target:

flip_gen = mytransforms.RandomFlipGenerator()
mytransforms.Compose([
    MyJointOp(), # take all inputs/targets
    [transforms.ColorAugmentation(), None], # Color augmentation in input, no operation in target
    flip_gen, # get a random seed for the flip transform and returns the identity
    [transforms.RandomFlip(flip_gen), transforms.RandomFlip(flip_gen)], # apply the same flip to input and target
    ...
])

[bodokaiser]

@fmassa Do you plan to merge your implementation into torchvision?

[fmassa]

@bodokaiser for the moment I'll only merge the part that separates the random parameters from the transforms, so that you can apply the same random transform to different inputs / targets.
We decided that my proposal brings too much complexity, and that it's up to the user to subclass their dataset and add the random transforms that you want.