Hi Everyone, as promised, in this thread we will answer the main questions that were left unanswered during the online meetup:
Q: Does the number of possible submissions per day depends on the number of members in the team?
A: No, each team can submit up to 2 successful submissions per day, regardless of the number of team members. Please note that CodaLab Submission mechanism will not enforce this limit automatically, each team must make sure they adhere to the per-team submission limits (2 entries per day per team during stage 1).
Q: How can we create a team?
A: In Codalab you have a tab "Team". For more instructions go to the link in the end or contact us.
https://github.com/codalab/codalab-competitions/wiki/User_Teams
Q: What is the evaluation score?
A: The evaluation score is ROC AUC.
Q: What the participants need to submit?
A: Participants are asked to give a probability score for each segment in the provided test set.
Q: How is the test performed? Do you share the test set or run the submitted models yourself?
A: The test set is available to you. You only need to submit the predicted probabilities in a csv file. In the end, the winners will need to send their final models for a check to be eligible for the prize.
Q: What is the competition_path that the notebook asks to give after mounting the drive?
A: This is the path to the folder that the data is in. For example, the full path can be ‘/content/gdrive/My Drive/Data’
Then copy only the extension ‘My Drive/Data’ to the "competition_path".
Q: Is the center of mass an average or a vector with the size of the X-axis?
A: A vector with the same size as the X-axis (32 * 1)
Q: Is there only one human/animal in each segment?
A: Each segment is labeled with one target type. It is possible that in some segments there will be an animal and human if, for example, an animal passed for a short time at the same time the human was passing by.
Q: Regarding plants - why would there be any doppler shift?
A: For example, if the weather is windy The plants can move with the wind.
Q: Does the struct contain the radar parameters?
A: No
Q: Does Segment ID == Track ID?
A: No, Tracks can have one or more segments.
Q: Does the test set have similar statistics of the background?
A: Not necessarily.
Q: Can you explain about High SNR and Low SNR?
A: Signal-to-Noise ratio (abbreviated SNR or S/N) is a measure used in science and engineering that compares the level of the desired signal to the level of background noise. SNR is defined as the ratio of signal power to the noise power, often expressed in decibels. A ratio higher than 1:1 (greater than 0 dB) indicates more signal than noise.
Q: What do you mean by “Auxiliary data sets”?
A: The Auxiliary data is not a "natural" data.
Synthetic low SNR segments that were created by transforming the high SNR signals from the train set.
“Background” segments – Segments that were recorded by a sensor in parallel to segments with tracks but at a different range. These segments contain the recorded “noise”. Each segment also contains a field mapping to the original High or Low SNR track id.
“Experiment” locations – In these locations, only humans were recorded in a controlled environment, which doesn't necessarily reflect a “natural” recording.
Q: If I got it right, 5566 (or something like that) * 128 * 32. i.e. each sample is 32*128, and there are ~5K of those?
A: There are 6656 segments in the training set, in the Auxiliary set there are 50k segments approximately in each set (more than 100k segments).
Q: What information we don't have in the test set that is available in the training set?
A: The metadata of the training and auxiliary data sets has also: track_id, geolocation_type, geolocation_id, sensor_id, date_index, target_type.
Q: What's the rationale for the following preprocessing step in the baseline model: "The preprocess sets the maximum value of each I/Q matrix in-place of the values at the center of the target's mass"?
A: The purpose of switching the values of the center of the target's mass in the I/Q matrix with the max value of the matrix is to enhance this part of the matrix and focus the model on this part. Different alternatives, such as adding a binary channel to the matrix, containing only the center of mass data, can be also considered.
Q: Is the training set balanced?
A: the training set is not balanced by the target type. you can see that and more in the descriptive statistics notebook: https://colab.research.google.com/drive/11Lzihg2vKIbo4KAIIJxW5CRZIncoWgtL?usp=sharing
Q: How did you deal with the imbalanced data?
A: We used the Auxiliary data set. You can see in the Baseline Model notebook - https://colab.research.google.com/drive/1SkoA3wQ_3cYtxt3P5pF-DSB-MFGGRQnN?usp=sharing
Q: Is there a label for background only?
A: Yes "empty". All the background segments are in the 'MAFAT RADAR Challenge - Auxiliary Background(empty) Set' files.
Q: Is there a center of mass given for the test set?
A: Yes it is the "doppler_burst"
Q: Is there a reason to believe that geographical type can influence the signal?
A: Yes, the terrain can be different, the weather, etc.
Q: When you say absolute, do you mean for the complex number?
A: Yes.
Q: Is there a public and private test set?
A: Yes. This is a two-stage competition.
Q: The script that converts the data to the spectrogram, is it the verbal that you had in the slides, or is there a version in code (python)?
A: The code in python and the slides are the same. Notice that the mean of the I/Q matrix in the competition dataset is 0 and therefore it is being ignored in the python code.
Tsabar
(MAFAT Challenge Team)
Q&A Meetup - Technical
Q: Could you give some details about the performance of human operators analyzing these spectrograms? How do they do it? What's the accuracy/ miss percentages?
A: People do not analyze the Spectrograms (as images) to classify the signals. In some radar systems, the raw signal is converted to an audio signal where an operator is supposed to distinguish between the different target types by listening to the audible characteristics of the signal. The accuracy of this technique, by a well-trained operator, is estimated to be 60-70% at best.
Q: Track correlation is a tracker level phenomenon. Does your tracker deal with that to an extent? Should we expect a high correlation?
A: As mentioned in the webinar, a high correlation may exist between different data segments of the same track. However, we strongly advise NOT to make any use of such correlation, if it exists, as the testing set will NOT display such correlation.
Q: Is the noise approximately independent over time?
A: It is not clear what the term ‘independent’ refers to. It can be assumed that there is a high correlation of noise level within different segments of the same track.
Q: How did you calculate the object's center of mass?
A: The object’s center of mass is calculated based on the radar tracker which estimates the target overall velocity. This estimation has an error and therefore should be used as an ‘assisting information’ and not be taken as an absolute true value.
Q: Do we see doppler shifts in the background, isn't it static mostly?
A: Background objects are not necessarily static. Ground and buildings are static but trees, grass and alike are not static and therefore show in the radar doppler signal.
Q: Presented images with "spikes" due to limb movement. Could you spot where these spikes are in the spectrogram?
A: In some of the spectrograms (mainly with high SNR) such spikes are clearly shown and can be easily spotted.
Q: Can you show us some challenging spectrogram images? From the example, it looks like it should be easy to distinguish between humans and animals.
A: The examples that were presented during the meetup represent an easy case. In most cases, this is much more difficult, especially when the target radar return has low power relative to the noise & clutter (you can look at segments with low SNR in the training set).
Q: In places where there is more than one radar - do they interfere with each other?
A: Interference between different radars should not be considered.
Q: You've mentioned that the movement affects linearly opposite (e.g. forward/backward), can you show how it looks in a spectrogram?
A: If we assume that a certain target at a given speed towards the radar shows in the spectrogram at velocity (or Doppler index) 12, if it had been moving away from the radar it would have shown at doppler index 116 (128 – 12).
Q: Do the examples include segments with doppler ambiguity?
A: The examples included in the data set can be assumed to have no Doppler ambiguity.
Q: Are there only two directions of the target movement?
A: Only radial velocity is expressed in a Doppler shift. Therefore, there are only two relevant motion directions (in respect to the Doppler shift) either towards the radar or away from it.
Q: Is a change of direction possible in observations?
A: Yes, a target can change its relative velocity towards the radar during the timeframe of a single segment or a track.
Q: Can you tell if it is positive or negative based on the spectrogram image?
A: ‘Center of mass’ velocity indicates whether the velocity is positive or negative. If the center of mass Doppler value is equal or below 64 it means the object velocity is positive (i.e. object range decreases) and if it is above 64, object velocity is defined as negative (i.e. object range increases).
Q: What happens when the object moves to the sides?
A: Lateral motion has no impact on the Doppler shift and therefore has no reflection in the data.
Q: The difference between 'clutter' and 'noise' - is that clutter is explainable?
A: ‘Noise’ refers to an undesired signal that is being originated from the radar own electronics circuitry. ‘Clutter’ refers to an undesired signal that is being originated from external sources other than the target of interest. For example, if the target of interest is a person then any signal that is being originated from swaying vegetation in the vicinity of the person would be included in the reference to ‘Clutter’.
Q: Can you explain about clutter and a human spectrogram?
A: This has already been explained in other Q&A as well. ‘Clutter’ refers to an undesired signal that is being originated from external sources other than the target of interest. Therefore, any other external Doppler signal source other than a ‘human’ target (assuming a human target is tracked) would be called ‘Clutter’. The spectrogram would include both the human doppler return and the clutter doppler return.
Q: How can I visualize micro-doppler for moving and non-moving parts? Do you have a picture of that? (for example a moving car without moving wheels and a moving car with moving wheels).
A: A moving car at a constant velocity without a moving wheel will show as a single frequency in the micro-doppler. Once the wheels are added, their rotation speed will add a more complex ‘signature’ to the micro-doppler signal. Yet, the vehicle body is expected to generate a strong signal and the wheels a much weaker one.
Q: Is the moving animal has almost the same DB across the 32 slices?
A: Target signal power fluctuations are dominated by changes in target radar cross section due to target changing geometry along time. These fluctuations could be quite significant and therefore it cannot be assumed that signal power is constant across a data set (32 time slices).
Q: Does clutter and animal DB look exactly the same, can you explain the difference?
A: ‘Clutter’ refers to an undesired signal that is being originated from external sources other than the target of interest. Therefore, any other external Doppler signal source other than an ‘animal’ (assuming an animal target is tracked) would be called ‘Clutter’. Clutter objects in such a case could be swaying trees or vegetation and they could (and mostly do) have a different look of their spectrogram than animals.
Q: The y-axis you said we see frequencies. Does that mean that we're seeing return signals from different sending times?
A: No, different return frequencies in the Y axis occur at the same time. They originate from returns that have different relative velocity towards the radar. For example, a person walking towards the radar has at a certain moment one leg moving towards the radar and a second leg moving away from the radar. Both legs' radar return would occur at the same time but will show at different frequencies in the micro-doppler.
Q: We got 128*32 samples, are the 4096 samples continuous?
A: Each set of 128 samples is continuous. You cannot assume continuity between different sets of 128 samples.
Q: Do you think there could be outliers in the data while extraction or representation in the CSV?
A: We did our best to prevent such outliers in the data. However, a careful approach should be taken and up to 1% of tagging error could be assumed.
Q: When you say different sensors, you mean different specs or different locations?
A: Different sensors refer to different specs. However, the data from all different sensors is normalized.
Q: Is all the data from the same radar type?
A: No. there is data from several radar types.
Q: Does the radar change its transmission when tracking a target, except for direction?
A: The radar waveform (i.e. ‘transmission’) could be changed along the track. It could be assumed constant only along each 128 samples.
Tsabar
(MAFAT Challenge Team)
Thanks- re:
Q: When you say different sensors, you mean different specs or different locations?
A: Different sensors refer to different specs. However, the data from all different sensors is normalized.
Normalized only in power or normalized as in we may assume that for all sensors/all segments spectrograms the x-axis represent the same time period (32 divisions), and the y-axis represents the same velocity range (128 divisions)?
Posted by: Pagey @ July 26, 2020, 6 p.m.Normalization of all axis values.
Tsabar
(MAFAT Challenge Team)
A clarification:
Q: Does the radar change its transmission when tracking a target, except for direction?
A: The radar waveform (i.e. ‘transmission’) could be changed along the track. It could be assumed constant only along each 128 samples.
Does that mean that it can also change within each segment?
Thanks.
Posted by: om_the_tortoise @ Aug. 1, 2020, 7:02 a.m.Hi,
a segment is a part of a track, therefore it can also change within a segment. The change can happen when moving along the "slow-time" axis (x-axis).
Tsabar
(MAFAT Challlenge Team)