Notebook status: reading notes, not a replication or a new empirical result. I will update this page as I check the original papers and implementations.
Why three passes?
When reading a foundational paper, it is tempting to highlight every paragraph on the first attempt. That creates a lot of marked text but not necessarily a working model of the method. I am testing a three-pass routine instead: identify the paper’s argument, reconstruct the learning objective, and then inspect the details that affect reproducibility.
The method below is especially useful for Word2Vec because its central idea is simple to state, while training choices and evaluation details can be easy to gloss over.
Pass 1 — map the paper
The first pass is deliberately quick. I write down the problem, the central proposal, the evidence offered, and the assumptions that seem to matter. I avoid trying to understand every equation immediately.
- Problem: learn dense word representations from local context.
- Core idea: use a prediction task as a training signal for vector representations.
- Evidence: inspect which intrinsic or downstream evaluations are reported and what they actually establish.
- Unresolved: identify implementation choices that might change the result.
Pass 2 — understand the objective
Word2Vec is a family of related approaches rather than a single, fully specified recipe. In skip-gram, a center word is used to predict surrounding context words. In continuous bag-of-words (CBOW), context words are used to predict the center word.
For a skip-gram model with a softmax output, the probability of context word \(o\) given center word \(c\) can be written as:
The expression is a useful reference point, but it does not describe every practical training setup. Negative sampling, for example, changes the optimization objective and avoids normalizing over the entire vocabulary at each update. I need to keep the paper’s stated objective separate from the implementation’s efficiency choices.
Pass 3 — reproduce the claims
The final pass turns reading into a checklist. I record vocabulary construction, tokenization, context-window policy, subsampling, vector dimensions, training duration, and the evaluation protocol. If a paper leaves a detail unspecified, I mark it as unknown rather than silently filling it in.
Paper:
Task and baseline:
Corpus and preprocessing:
Architecture / objective:
Hyperparameters:
Evaluation protocol:
Main result:
Limitations and unknowns:A reproduction attempt should start with a small sanity check: confirm that loss behaves as expected, inspect a few nearest neighbors, and verify that the evaluation code matches the intended metric. Nearest-neighbor examples are diagnostic, not a substitute for quantitative evaluation.
Questions to keep open
- Which conclusions depend on the corpus and preprocessing rather than the embedding objective itself?
- How sensitive are results to context-window size, subsampling, and negative-sampling settings?
- Which intrinsic benchmarks correlate with the downstream task I actually care about?
- How should these lessons change when text is code-switched or written with inconsistent orthography, as in Algerian Darija?
References
- Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. arXiv:1301.3781. https://arxiv.org/abs/1301.3781.
- Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013). Distributed Representations of Words and Phrases and their Compositionality. NeurIPS. arXiv:1310.4546.
Arabic summary
هذه ملاحظات أولية حول طريقة قراءة أوراق Word2Vec عبر ثلاث مراحل: فهم المشكلة والفكرة العامة، ثم إعادة بناء هدف التدريب، وأخيرًا تدوين تفاصيل التنفيذ والتقييم اللازمة لإعادة التجربة.
الفكرة الأساسية في Skip-gram هي التنبؤ بالكلمات المحيطة انطلاقًا من الكلمة المركزية، بينما يحاول CBOW التنبؤ بالكلمة المركزية انطلاقًا من السياق. يجب التمييز بين الصياغة النظرية وخيارات التنفيذ مثل Negative Sampling، مع تسجيل التفاصيل غير المذكورة بدل افتراضها.