Please post any questions related to data in this thread.
Posted by: Shubhankar @ April 16, 2021, 11:55 a.m.Hi.
During EDA, we found the following images to be a bit noisy:
* https://i.ibb.co/1JhFtRD/image.png
* https://ibb.co/Dr1FFb4
* https://ibb.co/WGpwzND
* https://ibb.co/rfWtJy7
Are these indeed noise? We understand that the segmentation maps can be all black and white in portions but what about the case where VV and VH images are all white but still there's some water level or flood level indicated for them in the maps?
Posted by: sayak @ April 17, 2021, 1:31 a.m.Hi.
* https://i.ibb.co/1JhFtRD/image.png : Can you explain what do you mean by noise here. The images are not RGB images so they may look noisy in RGB channels. Also the whole raw images are slightly tilted at an angle and hence the cropped results for the corner images tend to have black edges. However, this shouldn't cause any major issues for model training.
* https://ibb.co/Dr1FFb4 : Can you tell me image ID <*>_x_?_y_?.png . I can look that up in the raw data.
* https://ibb.co/WGpwzND
* https://ibb.co/rfWtJy7
Thanks!
Posted by: Shubhankar @ April 17, 2021, 1:58 a.m.> Can you explain what do you mean by noise here. The images are not RGB images so they may look noisy in RGB channels. Also the whole raw images are slightly tilted at an angle and hence the cropped results for the corner images tend to have black edges. However, this shouldn't cause any major issues for model training.
The upper portion of the image is white but still, there's a water body reported. The image is an RGB composite, so not sure what did you mean by "The images are not RGB images so they may look noisy in RGB channels."
This is the ID: northal_20191004t234700. Sorry I could not query it with the exact name. But with the ID I think you could still query the dataset with a reduced space.
Posted by: sayak @ April 17, 2021, 2:16 a.m.* The upper portion of the image is white but still, there's a water body reported. The image is an RGB composite, so not sure what did you mean by "The images are not RGB images so they may look noisy in RGB channels."
As I mentioned before, the actual images are processed from VV and VH (polarization) bands (you can learn more about the acquisition process here: https://sentinel.esa.int/web/sentinel/user-guides/sentinel-1-sar/acquisition-modes/ and generation process here: https://nasa-impact.github.io/etci2021/ ) which are slightly rotated and since we use 256x256 sliding window to crop these images their corresponding bitmaps may have slight overhangs, which are generated using another process, but this is only the case at the edges of processed VV and VH images.
In total there are 6-channels (3 for VV and 3 for VH)
* This is the ID: northal_20191004t234700. Sorry I could not query it with the exact name. But with the ID I think you could still query the dataset with a reduced space.
These images are also the edge cases where the sliding crop window was able to capture only a tiny portion of the VV/VH image. There are a few images like this with corresponding flood labels completely blacked out (no flood info present) however there may be water body label for the same since they have been acquired using different method. If you find any images which are not at the edges and can give me the exact coordinates I can try to look them up.
Thanks!
Posted by: Shubhankar @ April 17, 2021, 2:48 a.m.> If you find any images which are not at the edges and can give me the exact coordinates I can try to look them up.
I genuinely think you could narrow down your search with the ID I gave. Essentially 1008 images. But there are other similar IDs as well for which we observed similar things. Should we just treat them as noise?
Posted by: sayak @ April 17, 2021, 2:52 a.m.The actual image looks like https://ibb.co/jWFW1sv . I hope you get a better idea of sliding a crop window on this image. Yes, you could treat them as noise. Thanks!
Definitely yes. Thank you for all your help. I think you could enlist these pointers inside the homepage under FAQ so as to reduce a bit of load.
Posted by: sayak @ April 17, 2021, 3:13 a.m.Yes, I am collating a list of questions for FAQs as they come. I will create the section soon. Thanks for your questions.
Posted by: Shubhankar @ April 17, 2021, 3:25 a.m.Hi,
We were just wondering how the border tiles are being treated during testing. Are they considered for IOU calculation or discarded?
https://ibb.co/82vQPVv
This is an example of tiles that we came across, which seem to have no valid image pixels (white pixels) but a flood mask available. Also the flood mask here seems rather an artifact than an actual flood (rectangular shape).
Any feedback would be highly welcome.
This is a known phenomenon it seems. If you check out the earlier discussions in the post, you might get a few cues.
Posted by: sayak @ April 23, 2021, 7:12 a.m.Thanks for the reply. I saw that you already reported this and similar effect earlier. However, it is not yet clear to us how this is actually handled during testing. Such effects are also present in the current test data and we are wondering if we are supposed to predict this as well or can we simply discard them?
Posted by: mwlan @ April 23, 2021, 7:22 a.m.Discarding them is not an option I guess since that would reduce the number of samples from the expected array which is supposed to have a shape of [10400, 256, 256]. If this shape is not respected the submission portal will yield an error.
Posted by: sayak @ April 23, 2021, 7:30 a.m.