OpenKBP - 2020 AAPM Grand Challenge Forum

Go back to competition Back to thread list Post in this thread

> bug in evaluation ?

dose_score_vec[idx] = np.sum(np.abs(reference_dose - new_dose)) / np.sum(self.possible_dose_mask)

from this line, seems dose can exists beyond dose mask ? if not , the calculation should be

dose_score_vec[idx] = np.sum(np.abs(reference_dose - new_dose) * self.possible_dose_mask) / np.sum(self.possible_dose_mask)

Posted by: argman @ April 18, 2020, 2:33 a.m.

That was coded intentionally, but I understand why it looks like a bug. We want to encourage everyone to submit a dose that is already masked by the possible dose mask in which case the evaluation is equivalent to your suggestion. If you don't mask the dose you're effectively penalized. We want to encourage masking because it will keep the size of the files down and make any post-competition analysis much smoother.

Posted by: OpenKBP @ April 20, 2020, 6:37 p.m.

I think its an error, if the ground truth dose exists where mask = 0, this will also penalize the submission, so leaderboard result is under-estimated.

Posted by: argman @ April 21, 2020, 1:57 a.m.

In the preprocessing we ensure that ground truth dose is always equal to 0 where the mask is equal to 0. Otherwise if the truth dose did exist where the mask = 0, I would agree with your comment.

Posted by: OpenKBP @ April 21, 2020, 4:18 p.m.

I think you need to check the dataset. in some patients dose does exists where mask=0

Posted by: argman @ April 22, 2020, 1:42 a.m.

That's odd. I'll look into this tomorrow. Can you provide me with some patient numbers where dose exists outside the mask?

Posted by: OpenKBP @ April 22, 2020, 1:51 a.m.

I just ran a script to validate that there is no dose outside any possible dose masks. You can run it yourself by inserting it at line 30 in the script. Please let me know if you have an example patient where the dose exists outside the dose mask, or if you find a bug in this short script.

data_loader_train = DataLoader(training_paths, batch_size=1)
patient_max_dose_list = []
for i in tqdm.tqdm(range(num_train_pats)):
pat = data_loader_train.get_batch(i)
dose = pat['dose'].flatten()
mask = pat['possible_dose_mask'].flatten()
max_dose_outside_mask = dose[mask <= 0.5].max()
print('The max dose outside all possible dose masks in the training set is {}'.format(max(patient_max_dose_list)))

Posted by: OpenKBP @ April 22, 2020, 3:55 p.m.

Coda lab formatting seems to get rid of tabs for the for loop I posted. There should be tabs from where the for loop starts to the line just before the print statement.

Posted by: OpenKBP @ April 22, 2020, 3:56 p.m.

thanks, i checked the dataset and found the masks and dose are right, its my mistake

Posted by: argman @ April 23, 2020, 6:18 a.m.
Post in this thread