https://github.com/ababier/open-kbp/blob/3c91f1214b112005765cb0e46f043b562d43da50/provided_code/dose_evaluation_class.py#L72
dose_score_vec[idx] = np.sum(np.abs(reference_dose - new_dose)) / np.sum(self.possible_dose_mask)
from this line, seems dose can exists beyond dose mask ? if not , the calculation should be
dose_score_vec[idx] = np.sum(np.abs(reference_dose - new_dose) * self.possible_dose_mask) / np.sum(self.possible_dose_mask)
Posted by: argman @ April 18, 2020, 2:33 a.m.That was coded intentionally, but I understand why it looks like a bug. We want to encourage everyone to submit a dose that is already masked by the possible dose mask in which case the evaluation is equivalent to your suggestion. If you don't mask the dose you're effectively penalized. We want to encourage masking because it will keep the size of the files down and make any post-competition analysis much smoother.
Posted by: OpenKBP @ April 20, 2020, 6:37 p.m.I think its an error, if the ground truth dose exists where mask = 0, this will also penalize the submission, so leaderboard result is under-estimated.
Posted by: argman @ April 21, 2020, 1:57 a.m.In the preprocessing we ensure that ground truth dose is always equal to 0 where the mask is equal to 0. Otherwise if the truth dose did exist where the mask = 0, I would agree with your comment.
Posted by: OpenKBP @ April 21, 2020, 4:18 p.m.I think you need to check the dataset. in some patients dose does exists where mask=0
Posted by: argman @ April 22, 2020, 1:42 a.m.That's odd. I'll look into this tomorrow. Can you provide me with some patient numbers where dose exists outside the mask?
Posted by: OpenKBP @ April 22, 2020, 1:51 a.m.I just ran a script to validate that there is no dose outside any possible dose masks. You can run it yourself by inserting it at line 30 in the main.py script. Please let me know if you have an example patient where the dose exists outside the dose mask, or if you find a bug in this short script.
data_loader_train = DataLoader(training_paths, batch_size=1)
patient_max_dose_list = []
for i in tqdm.tqdm(range(num_train_pats)):
pat = data_loader_train.get_batch(i)
dose = pat['dose'].flatten()
mask = pat['possible_dose_mask'].flatten()
max_dose_outside_mask = dose[mask <= 0.5].max()
patient_max_dose_list.extend([max_dose_outside_mask])
print('The max dose outside all possible dose masks in the training set is {}'.format(max(patient_max_dose_list)))
Coda lab formatting seems to get rid of tabs for the for loop I posted. There should be tabs from where the for loop starts to the line just before the print statement.
Posted by: OpenKBP @ April 22, 2020, 3:56 p.m.thanks, i checked the dataset and found the masks and dose are right, its my mistake
Posted by: argman @ April 23, 2020, 6:18 a.m.