Join Our 5-Week ML/AI Engineer Interview Bootcamp 🚀 led by ML Tech Leads at FAANGs
71. Positional encoding
GeneralImplement sinusoidal positional encoding so you can add position information to token embeddings in an NLP model. You’ll generate a matrix where each position has a deterministic encoding based on sine and cosine waves.
The encoding is defined as:
where (pos) is the token index (starting at 0), (i) is the dimension index, and (d) is the model dimension.
Requirements
Implement the function
Rules:
- Use the sinusoidal formula above with positions
pos = 0..seq_len-1. - Even dimensions use
sin, odd dimensions usecos. - Return the result as a NumPy array.
- Use only NumPy and Python built-in libraries (no PyTorch/TensorFlow).
- Aim for a vectorized NumPy implementation (avoid deeply nested loops if you can).
Example
Output:
| Argument | Type |
|---|---|
| d_model | int |
| seq_len | int |
| Return Name | Type |
|---|---|
| value | np.ndarray |
Constraints
Return NumPy array
Use NumPy only; no torch/tensorflow
Vectorize with broadcasting; avoid nested loops
Hint 1
Create a (seq_len, 1) array of positions with np.arange(seq_len)[:, None] so it broadcasts across dimensions.
Hint 2
Compute the scaling term only for even dimensions: i = np.arange(0, d_model, 2) and use np.exp(-np.log(10000.0) * i / d_model) to match 1 / 10000^(2i/d).
Hint 3
Fill a (seq_len, d_model) matrix by slicing: pe[:, 0::2] = sin(pos * div_term) and pe[:, 1::2] = cos(pos * div_term), then return pe.
Roles
Companies
GeneralLevels
Tags
Python Editor