Competition

Evaluation Criteria

Different from the current layout estimation approaches, we only evaluate the performance on the visible layout, i.e., ignore the occluded layout by the front wall. For this task, we use the following three metrics for room layout estimation evaluation, and the final metric is the mean of these three metrics.

• Junction: We consider the predicted corner is correct only if the distance between this junction and its nearest ground-truth is within a threshold. The threshold is set to 5, 10, and 20 pixels.
• Wireframe: Similarly, we consider the predicted line segment is correct only if the distance between this line segment and its nearest ground-truth is within a threshold. The threshold is set to 5, 10, and 20 pixels.
• Plane: We consider the predicted plane is correct only if the intersection-over-union between this plane and its nearest ground-truth is larger than 0.5.

The evaluation code is available at Github.

Annotation Format

We provide the amodal layout annotation as an ordered list of 2D positions of the junctions. In our dataset, the cameras of the panoramas are aligned with the gravity direction. Thus, a pair of ceiling-wall and floor-wall junctions share the same x-axis coordinates.

Submission Format

The predicted layout should be represented by an Nx2 integer array with a 512x1024 image resolution and only contains the visible layout (shown in the figure above). Thus, the x coordinates of the layout is a monotonically non-decreasing sequence. To submit your results to the leaderboard, you have to construct a submission zip file containing prediction text files (".txt") with the same name as the corresponding testing images:

$ zip -rj submission.zip folder-of-your-results

General Rules

• Each entry must be associated with a team and provide its affiliation.
• The limit in the number of submissions is per team and not per user. We define a team as a group of people that would coauthor the final publication. Please follow the instruction to create the competition team.
• You are allowed to use other annotations in the Structured3D dataset. If you use additional annotations (e.g., depth, normal) in the Structured3D dataset, please specify in the "method description" when submitting your results.
• The best entry of each team will be public in the leaderboard at all times.
• We reserve the right to remove one of the entry methods to the competition when the entry is incomplete or violates the rules.

Dataset

The Structured3D dataset is released under the Structured3D Terms of Use. Please cite the Structured3D if you participate in the challenge or find this dataset helpful in your research:

@inproceedings{Structured3D,
  title     = {Structured3D: A Large Photo-realistic Dataset for Structured 3D Modeling},
  author    = {Jia Zheng and Junfei Zhang and Jing Li and Rui Tang and Shenghua Gao and Zihan Zhou},
  booktitle = {Proceedings of The European Conference on Computer Vision (ECCV)}),
  year      = {2020}
}