Play Abdur Rahim’s 60-second demo first. Ask: what must be stored to jump to the right moment? These are the developer’s example queries, not results reproduced in class. The app is a native Apple Silicon Mac application using llama.cpp and Metal, not WebGPU. Our browser lab is a separate implementation of related ideas.

CLIP connected image and text representations. EmbeddingGemma 2 also supports audio and video inputs; it is a different pretrained model, not an extra modality added to our CLIP checkpoint.

Diagram is a simplified source-grounded view. DigUp stores L2-normalized 768-dimensional vectors as float16 in SQLite, along with segment kind and location. Photos use visual content; screenshots also get OCR for keyword search. PDFs can supply page images and text passages. Audio uses overlapping 30-second windows; video uses sparse frames, shot selection and audio windows.

Indexing runs the pretrained model; it does not train it on your folder. Files must be indexed before semantic search can retrieve them. The app incrementally handles changes. A miss may be caused by unreadable or unsampled content, not just the embedding.

This is a teaching skeleton, not DigUp source. extract, encode_pieces and save_json are application functions: extraction supplies text passages, decoded images, audio windows or video frames; encoding calls the model’s modality-specific adapter. They are named abstractions, not claimed library APIs. Cache model revision, preprocessing settings and an input-content hash so changed content can be re-embedded. Metadata values above are illustrative.

For text, the official model card uses SentenceTransformer("google/embeddinggemma-2"), model.encode(documents, prompt_name="Document", normalize_embeddings=True). Queries use SearchQuery. Images/audio are sent through their media input paths without text task prefixes. Use the same model, revision, dimensions and compatible preprocessing for index and query. See PRACTICAL.md for a concrete text-only index script and the contract for adding other modalities.

Every number and filename on this slide is illustrative. All four displayed vectors have norm 1. Multiplication is the same as CLIP’s unit-vector comparison, now over stored pieces. The real app uses 768 dimensions and scores the stored matrix in blocks. These three toy cosine scores are not DigUp’s final ranking scores: the next slide adds its documented calibration, keyword matching and file grouping. A cosine is a similarity score, not a probability.

The pinned Searcher.swift computes cosine similarity, subtracts modality means and divides centered scores by a shared scale with a floor. It selects the best semantic segment per file. FTS5 searches file/folder names, document text and OCR. Files qualify through semantic salience or enough query-word coverage.

For ordinary non-code queries, score = max(z,0) + 2 × coverage^1.5 + a file-kind bonus when applicable. Lookup-like queries halve the semantic weight and double the word weight; code uses a smaller word weight. This is a concise description of that revision, not a universal formula for embedding search. Exact IDs and dates are useful tests of the lexical route.

This is the semantic baseline. The query helper in the companion script calls the official SearchQuery prompt. Keep raw cosine scores; they are not confidence probabilities. A production system can add DigUp’s lexical index, calibration and file grouping. For larger corpora, an approximate-nearest-neighbor index trades some recall for search cost, but it is unnecessary for the first small example.

Check whether the correct source is in the top five for questions you wrote before tuning. Include keyword-sensitive examples and unanswerable questions. A stored location must be valid even if its semantic score is high. The companion script runs on text pieces; a media extractor must be implemented before claiming this searches an arbitrary folder.

These are schematic routes based on documented DigUp behavior. Source-code search is opt-in and has its own index; explicit identifier queries also use definition lookup. Semantic matches generally open the best segment; exact word hits may determine a more precise location. Sparse video frames can miss a brief event; index completeness and source metadata are part of retrieval quality.

Class question: if we change a lecture slide, what must be refreshed? Answer: its extracted content and embedding, while preserving the correct lecture/slide link. Our existing course-search lab illustrates the same pattern; it is a separate app, not a DigUp integration. No generative answer model is needed to show cited excerpts.

Play Shahir’s 26-second video. The project site reports 12.6 seconds and calls it 288× real time; rounded figures need not divide to that exact factor, so this slide uses “about 13 seconds.” These are the developer’s measurements, not a course benchmark. Author replies report 5.18% WER on LibriSpeech test-clean and about 11.5–11.8% on the harder test set. The app uses the Apple Neural Engine and is not a WebGPU demo.

Ask whether finding a recording “about sleep” is enough to recover its exact words. Retrieval can tolerate paraphrase; transcription must preserve lexical detail and time order.

The public description says the audio encoder is unchanged, a small reader maps short audio slices into the model’s text space, nearest words are selected, and a tiny model adds punctuation/capitalization. This is a high-level diagram of those stated components. The exact slice duration, reader layers, alignment procedure and training loss are not specified in the public notes inspected on 13 October 2026.

Do not infer that an arbitrary 768-dimensional pooled embedding uniquely determines a transcript. Similar meanings can have different wording. We do not label this reader as CTC or an autoregressive decoder without implementation evidence.

All 2D vectors, word scores and the winning word on this slide are invented for teaching. Every vector is unit length, so a dot product equals cosine. The documented project selects the nearest of about 229,000 English words in the model’s text space. The exact distance function is not publicly specified; cosine/argmax here illustrates nearest-vector lookup, not a reproduced implementation or output.

The table contains candidate word representations, not the token-embedding matrix of a language model. The public notes do not specify how repeated slice predictions are aligned or merged. A tiny additional model restores punctuation and capitals.

Proposed reproducible teaching dataset, not the dataset claimed for Shahir’s reader. Speech Commands v0.02 contains short labelled spoken-word WAVs. Select the eight named word directories and retain the official validation/testing lists; other eligible files are training data. The speaker identifier before _nohash_ enables an explicit leakage check. Clips may be shorter than one second: preprocess consistently as mono 16 kHz and keep a record of any padding. Do not mix cropped portions of one recording across splits.

Compute one native 768D normalized EmbeddingGemma 2 output for each clip, with the model frozen. These semantic embeddings may lose phonetic distinctions, so success is an experiment to measure, not a promised result. Eight-way word accuracy is the appropriate first metric; it is not sentence-level WER.

Architecture chosen for this classroom experiment: Linear(768,128), ReLU, Linear(128,768), then L2 normalization. B is minibatch size. Each input coordinate is one feature; the drawing shows only three neurons from each layer and marks the omitted ones. The two linear layers have 197,504 trainable parameters including biases. Candidate words are encoded with the frozen text branch using a consistent Classification prompt and normalized. The matrix E is held fixed.

Multiplying normalized reader outputs by E transpose supplies cosine-based logits, divided by a positive temperature. A simpler Linear(768,8) classifier is also a valid baseline. This slide intentionally illustrates a text-vector readout to connect to CLIP and the author’s nearest-word idea. The author’s actual reader input dimensions and layers are not disclosed.

Three candidate words are shown only to make the arithmetic small; the proposed experiment has eight. Probabilities are illustrative and do not come from a trained model. With logits S = normalize(reader(X)) Eᵀ / tau and labels y, use cross_entropy(S, y). PyTorch performs log-softmax internally, so do not pass already-softmaxed probabilities to cross_entropy. The default mean reduction averages the minibatch.

This cross-entropy is our explicitly chosen word-classification objective. It is not claimed to be Shahir’s unpublished reader loss. Only reader parameters are in the optimizer. For head-only training, cached embeddings detach the base model entirely. Finetuning the encoder instead requires audio inputs, gradients through the model, a new optimizer scope, and rebuilding any affected index.

Imports, validated cache loading, shuffling, epochs and validation are in code/train_reader.py. The code is a teaching reader, not the speech app’s source. The runnable companion reports validation accuracy and selects the best validation epoch; test evaluation is separate. It accepts actual cached 768D features and does not generate fake embeddings if the cache is absent. The shape-and-gradient check uses random tensors and does not measure model quality.

Compare this reader with the zero-shot baseline X Eᵀ on the same held-out speakers. Better training loss alone is insufficient. The proposed learning rate, hidden width and temperature are starting choices to tune only on validation data.

LibriSpeech is a proposed next dataset, not a verified training source for Shahir’s model. It provides 16 kHz read English audio and transcripts under CC BY 4.0. The author names only openly licensed English speech and reports LibriSpeech evaluation. Do not convert that evaluation fact into a claim about their training corpus.

Two practical designs: keep a temporal feature sequence and train with an alignment-aware sequence objective such as CTC; or obtain word timings using an external forced aligner, train on aligned segments, and design segmentation/merging for inference. These are proposed options, not claims about the original implementation. A faithful reproduction would require the reader architecture, feature tap, objective, corpus, vocabulary and decoding details from the author. Score final transcripts by WER and measure end-to-end runtime on the same recordings.

The author reports training the reader in under two hours on eight H100 GPUs using openly licensed English speech; fast on-device inference does not mean this training ran on the laptop. The pretrained embedding model was not originally trained as this speech recognizer. The public method leaves the exact objective unspecified, so the diagram intentionally shows only the supported direction of learning.

Evaluate transcription using WER = (substitutions + deletions + insertions) / reference words, and runtime on the same audio and hardware. A good retrieval result and a good transcript require different evidence. The author acknowledges higher errors on their own speech. Download links lead to independent third-party Mac apps; the lecture itself does not install anything.