Computer Vision Interview Questions

Sample Answer

Skip connections let each residual block learn a residual function $\mathcal{F}(x) = H(x) - x$ instead of the full mapping $H(x)$, so the network only needs to learn small perturbations to the identity. This directly addresses the degradation problem: without skip connections, deeper plain networks suffer from optimization difficulty because gradients vanish or explode through long chains of nonlinear layers, making it harder to learn even an identity mapping. With the shortcut, gradients flow directly through the addition operation during backpropagation, providing a clean path that scales to hundreds of layers. You should also mention that this is distinct from the vanishing gradient problem solved by BatchNorm; the degradation problem is specifically about optimization landscape, not gradient magnitude.

Sample Answer

You could go with MobileNetV2 for its simpler inverted residual blocks with depthwise separable convolutions, or EfficientNet-B0 for its higher accuracy from compound scaling. MobileNetV2 wins here because its architecture uses uniform, hardware-friendly operations that map efficiently to fixed-function accelerators like Tesla's FSD chip, while EfficientNet-B0's squeeze-and-excitation blocks and variable resolution scaling introduce irregular memory access patterns that inflate real-world latency beyond what FLOPs alone predict. You should always benchmark actual inference time on the target hardware rather than relying on theoretical FLOP counts, since operations like SE blocks and swish activations can be disproportionately slow on specific accelerators. If accuracy is still insufficient, consider knowledge distillation from a larger EfficientNet teacher into the MobileNetV2 student.

Sample Answer

Let's reason through this step by step. A $1 \times 1$ convolution operates on a single spatial location, so its receptive field covers only one pixel per layer. This means you lose the ability to capture local spatial patterns like edges, textures, and shapes, which are fundamental to image classification. The network would need an impractical number of layers just to aggregate spatial context, and even then the learned features would be poor because early layers cannot detect basic visual structures. What you should suggest instead is the Inception-style factorization: replace a $3 \times 3$ conv with a $1 \times 1$ conv to reduce channel dimensionality, followed by a $3 \times 3$ conv, cutting compute by roughly $3\times$ to $9\times$ depending on the reduction ratio while preserving spatial feature extraction.

Sample Answer

You could go with a two-stage detector like Faster R-CNN for higher accuracy on small or occluded pedestrians, or a one-stage detector like YOLOv8 for lower latency. One-stage wins here because autonomous driving demands real-time inference (often under 30ms per frame), and modern one-stage detectors have largely closed the accuracy gap through improvements like Feature Pyramid Networks and advanced label assignment strategies. You should communicate to your team that the tradeoff is not purely speed vs. accuracy anymore: with proper anchor-free designs and strong augmentation, one-stage models achieve competitive mAP while being 3 to 5 times faster. If your system still misses hard cases, you can add a lightweight second-stage refinement head without adopting the full two-stage pipeline.

Sample Answer

Let's reason through this step by step. In a standard CNN backbone, early layers have high resolution but weak semantics, while deep layers have strong semantics but low resolution. FPN builds a top-down pathway that takes the semantically rich deep features and upsamples them, then fuses them with the corresponding high-resolution lateral connections via element-wise addition at each pyramid level. This gives every level both strong semantics and appropriate spatial detail. If you removed the top-down pathway, the lower pyramid levels would only have the raw backbone features, which lack the semantic context needed to distinguish small objects from background clutter. You would see a significant drop in AP for small objects because those detections rely on the fine-grained spatial information at early levels being enriched by the propagated high-level features.

Sample Answer

Start by recalling that panoptic segmentation requires labeling every pixel as either a "stuff" class (amorphous regions like sky or road) or a "thing" class (countable objects like cars or people) with unique instance IDs. Panoptic FPN uses a shared FPN backbone, then branches into two heads: a semantic segmentation head that predicts per-pixel stuff classes, and a Mask R-CNN style head that predicts instance masks for thing classes. The merging heuristic resolves conflicts by first ranking instance predictions by confidence score, then greedily assigning pixels to the highest-confidence instance, and finally filling any remaining unassigned pixels using the semantic segmentation output. You should mention that the quality metric is Panoptic Quality, defined as $PQ = \frac{\sum_{(p,g) \in TP} \text{IoU}(p,g)}{|TP| + \frac{1}{2}|FP| + \frac{1}{2}|FN|}$, which jointly captures recognition and segmentation quality.

Sample Answer

This question is checking whether you can diagnose why a standard architecture fails on specific data characteristics and propose targeted fixes rather than generic improvements. For small lesions, you should add attention gates to the skip connections so the decoder focuses on relevant spatial regions instead of propagating noisy low-level features. Replacing the standard cross-entropy loss with a combination of Dice loss and focal loss directly addresses class imbalance and penalizes poor performance on small objects: $$L = \lambda_1 \cdot L_{\text{focal}} + \lambda_2 \cdot (1 - \frac{2|P \cap G|}{|P| + |G|})$$. You could also incorporate atrous spatial pyramid pooling (ASPP) at the bottleneck to capture multi-scale context, and apply aggressive data augmentation like elastic deformations and color jittering that reflect real clinical variation.

Sample Answer

This question is checking whether you can bridge the gap between academic video models (which assume fixed-length clips) and production constraints like variable duration, throughput, and memory. You should propose a hierarchical approach: first, uniformly sample or segment the video into fixed-length clips (e.g., 2-second chunks), extract per-clip embeddings using a frozen or fine-tuned video encoder like ViViT or VideoMAE, then aggregate across clips using a lightweight temporal model such as a Transformer with learned positional encodings or a simple temporal attention pooling layer. For 10-minute videos, you need to be explicit about memory: mention that you would cap the number of sampled clips (e.g., $K=32$) and use sparse sampling strategies rather than processing every frame. At Meta's scale, you would also want the clip encoder to be distilled or quantized for serving, with the aggregation head being small enough to run online.

Sample Answer

The standard move is to use 3D convolutions (like C3D or I3D) to capture temporal information across frames. But here, computational cost matters because edge devices have strict latency and memory budgets. TSM works by shifting a portion of the channels along the temporal dimension before applying standard 2D convolutions, so you get temporal reasoning at zero extra FLOPs compared to a 2D ResNet. You shift, say, 1/8 of channels forward and 1/8 backward in time, then the subsequent 2D conv naturally mixes information across frames. This lets you reuse highly optimized 2D convolution kernels and pretrained ImageNet weights while still achieving accuracy competitive with I3D on benchmarks like Kinetics.

Sample Answer

The standard move is late fusion because it is modular, easier to pretrain each encoder independently, and computationally cheaper at inference since you can cache embeddings. But here, early fusion matters because tasks like visual question answering require fine-grained cross-modal reasoning where a word like 'left' must attend to specific spatial regions in the image, something late fusion handles poorly. Early fusion models (e.g., Flamingo's gated cross-attention or the cross-attention layers in BLIP-2) allow token-level interactions between modalities, yielding stronger performance on tasks requiring compositional understanding at the cost of higher compute and the inability to independently precompute embeddings. You should tell the interviewer that in practice, many state-of-the-art systems use a hybrid: separate encoders for initial feature extraction followed by cross-attention fusion layers, balancing modularity with expressive cross-modal interaction.

Sample Answer

Get this wrong in production and harmful content slips through moderation, creating real user safety and legal risks. The right call is to first audit per-class and per-demographic performance by building a stratified evaluation set that includes culturally specific edge cases, then measure embedding similarity distributions to see if nuanced categories cluster too closely in CLIP's shared space. You should address the gap by ensembling CLIP with a fine-tuned classifier trained on labeled examples of the failing categories, using prompt ensembling (multiple text descriptions per class averaged together), and potentially fine-tuning the model with a curated dataset using a contrastive or cross-entropy objective on the hard categories. Continuous human-in-the-loop feedback is essential here because the distribution of harmful content shifts over time.

Sample Answer

Get this wrong in production and you either flood human reviewers with false positives, destroying throughput, or you let defective products ship to customers. The right call is to first collect a sample of flagged images from the new center and visually compare them against your training distribution, looking for lighting changes, camera angle differences, new product SKUs, or background variation. Compute dataset-level statistics like pixel intensity histograms and feature-space distances (using embeddings from your backbone) between the training set and the new site's data to quantify the shift. Then apply targeted domain adaptation: fine-tune on a small labeled set from the new center (even 200 to 500 images helps), add domain-specific augmentations (color jitter, brightness shifts matching the new environment), and consider using a domain-adversarial training objective if the shift is severe. Set up continuous monitoring of per-site precision and recall with automated alerts so you catch future drift early.

Sample Answer

Random erasing and CutMix sound reasonable but break under medical imaging constraints, because occluding or mixing pathological regions can destroy the very signal the model needs to learn, especially with small lesions. Aggressive color jitter does not work well either because diagnostic features in X-rays depend on subtle grayscale intensity patterns that artificial color shifts can corrupt. That leaves you with domain-appropriate augmentations: mild geometric transforms (small rotations, flips consistent with anatomy), elastic deformations, and controlled intensity scaling. You should pair these with transfer learning from a model pretrained on a large radiographic corpus (or even ImageNet as a fallback) and consider self-supervised pretraining (e.g., MAE or SimCLR) on unlabeled X-rays if available, since with only 5,000 labels, representation quality matters more than augmentation volume.