train Panoptic Segmentation model on custom dataset

[Cyril9227]

❓ How to train Panoptic Segmentation on a custom dataset ?

Hello everyone,

My question is two-fold :

1. What is the expected Detectron2 format for Panoptic Segmentation ? In the PS paper, they are talking about a pixel-wise annotation (either stuff class or instance ID) but it seems that D2 requires bounding boxes as well.
2. Is it possible to train PS on a custom dataset with only pixel-wise annotations ?

Thanks,

Cyril

[ppwwyyxx]

At the moment we don't have instructions to train panoptic segmentation models on custom dataset. Our code now expects panoptic segmentation data in COCO format, which is unfortunately not well-documented.

[JavierClearImageAI]

At the moment we don't have instructions to train panoptic segmentation models on custom dataset. Our code now expects panoptic segmentation data in COCO format, which is unfortunately not well-documented.

Here the link to the COCO format ... then go to point 4: Panoptic Segmentation. I would also recommend to download the json annotations and mask labels in png format, have a look at them and create yours in the same format.

I am still on my way do it. After converting the annotations into the right format, how to register panoptic annotations for training? is it the same than with instance segmentation and object detection? Since we have now an extra directory with the masks, I am wondering if it will use the same code to register the annotations.

I guess the answer is no, since you created an specific script to do that:
coco_panoptic.py

detectron2.data.datasets.register_coco_panoptic

[mvdelt]

@JavierClearImageAI what is your way to make your custom panoptic segmentation annotations?

[JavierClearImageAI]

@JavierClearImageAI what is your way to make your custom panoptic segmentation annotations?

I ended up using detectron2.data.datasets.register_coco_panoptic_separated

I realized that using detectron2.data.datasets.register_coco_panoptic wasn't working for a custom dataset with new categories, since it registers a “standard” version of COCO panoptic.

To be honest, I don't really like that you have to provide the images, the masks, json for the instances and json for instance segmentation, since the instance segmentation and the semantic segmentation info could be automated by FAIR from the png binary masks... but you will have to do it if you want to register data from the jsons. Otherwise you might need to dive into the code and create your own dataloader, which might take you longer to code.

So, just follow the "coco standard panoptic format" for the panoptic coco-json and the "coco standard instance segmentation format" for the instances.

In my case "sem_seg_root" = "panoptic_root", since I use the same masks for both.

Summarizing:

1. You need to (convert your png image masks to contours)[https://github.com/convert mask binary image to polygon format  cocodataset/cocoapi#131] and create the coco instance segmentation annotations.
2. Then create the json for panoptic (check point 4 from "coco standard panoptic format")
3. Then you need to register your data using the afore mentioned method.
4. Configure the training as you did for custom instance segemtnation training
5. Train

PD: here my dataset registration code:

Method:

detectron2.data.datasets.register_coco_panoptic_separated(name, metadata, 
        image_root, panoptic_root, panoptic_json, sem_seg_root, instances_json).

My_code

register_coco_panoptic_separated(subset+"_"+tr1.backbone_config, {}, 
        join(tr1.dataset_dir,"images/"+subset), 
        join(tr1.dataset_dir,"masks/"+subset), join(anns_semantic_segm_dir, subset+".json"), 
        join(tr1.dataset_dir,"masks/"+subset), join(anns_instance_segm_dir, subset+".json"))

Appendix1: Panoptic annotations format

{
 "info": {...},
 "images": [
    {
      "id": 41,
      "width": 512,
      "height": 384,
      "file_name": "image_253.jpg",
      "date_captured": "2021-02-23 21:14:28.054086"
    }, ...
    ... 
  ],
  "annotations": [
    {
      "image_id": 1,
      "file_name": "image_213.png",
      "segments_info": [
        {
          "id": 1,
          "category_id": 1,
          "bbox": [
            0,
            207,
            511,
            176
          ],
          "iscrowd": 0
        }
      ]
    }, ...
    ...
    "categories": [{...},
      {...},
      ...]
}

[mvdelt]

@JavierClearImageAI Thank you for your detailed answer. So, are you using Panoptic FPN? (not panoptic deeplab?)
I guess register_coco_panoptic_separated is only for Panoptic FPN.

[JavierClearImageAI]

@JavierClearImageAI Thank you for your detailed answer. So, are you using Panoptic FPN? (not panoptic deeplab?)
I guess register_coco_panoptic_separated is only for Panoptic FPN.

I don't think detectron2 has panoptic deeplab backbone network implemented. It seems to be a Google research group backbone net. At least I couldn't see it in configuration options. They have 4 fpn backbone nets for panoptic only. Please let me know if I am wrong

[bowenc0221]

The Panoptic-DeepLab project in Detectron2 might be a good reference. The detectron2.data.datasets.register_coco_panoptic loads the raw panoptic annotation (from both png and json files), and you can process this data depending on the need of your model (example).

[tiusty]

@JavierClearImageAI
Just to clarify a few things:

When you say

I realized that using detectron2.data.datasets.register_coco_panoptic wasn't working for a custom dataset with new categories, since it registers a “standard” version of COCO panoptic.

Standard means that it is follows the same format from the COCO dataset and uses the same category ids that are already defined.

Custom means that is follows the same format from the COCO dataset but uses new category ids. Therefore if I want to train an already pre-trained PanopticFPN model from the model zoo with new categories, I would have to use a custom dataset in COCO format with the new category ids.

It is confusing to me because "standard" version would seem like it is referring just to the format, but it seems like it is also referring to the already defined categories.

Therefore to train a model with new categories we need to use the
register_coco_panoptic_separated
function like you mentioned.

Lastly when you say

So, just follow the "coco standard panoptic format" for the panoptic coco-json and the "coco standard instance segmentation format" for the instances.

I understand the coco standard panoptic format but is the coco standard instance segmentation format the object detection format from https://cocodataset.org/#format-data (i.e # 1 on the list)?

Thanks!