AI Knowledge Information 4 — Questions and Answers
Question 1: In information theory, what does 'entropy' measure in the context of a knowledge source?
- The average number of bits needed to encode information from a source (Correct answer)
- The accuracy of predictions made by a model
- The compression ratio of a knowledge base
- The redundancy within a neural network's parameters
Correct answer: The average number of bits needed to encode information from a source
Shannon entropy quantifies the average uncertainty or information content of a random variable, measuring how many bits are needed on average to represent outcomes.
Question 2: Which technique is used to evaluate whether a language model's generated answer is faithful to a retrieved source document?
- Perplexity scoring
- Faithfulness evaluation (e.g., using NLI models) (Correct answer)
- BLEU score computation
- Cosine similarity on token embeddings
Correct answer: Faithfulness evaluation (e.g., using NLI models)
Faithfulness evaluation uses natural language inference (NLI) models or dedicated metrics to check whether generated claims are logically entailed by the source document.
Question 3: What is 'sparse retrieval' in the context of document search for AI systems?
- Retrieving documents using term-frequency-based methods like BM25 (Correct answer)
- Retrieving only a small random sample of available documents
- Limiting retrieval to documents with fewer than 100 words
- Using a compressed vector index with fewer dimensions
Correct answer: Retrieving documents using term-frequency-based methods like BM25
Sparse retrieval relies on lexical matching using bag-of-words representations and algorithms like TF-IDF or BM25, contrasted with dense retrieval using neural embeddings.
Question 4: What challenge does 'knowledge staleness' pose for AI engineers deploying production systems?
- Models become computationally slower as knowledge ages
- The model's internal knowledge diverges from current real-world facts over time (Correct answer)
- Older training data causes overfitting to historical patterns
- Model weights degrade in precision due to floating-point drift
Correct answer: The model's internal knowledge diverges from current real-world facts over time
Knowledge staleness means a deployed model's parametric knowledge becomes outdated as the world changes, requiring mitigation via RAG, fine-tuning updates, or knowledge editing.
Question 5: Which approach best mitigates conflicting information when merging multiple knowledge sources?
- Always prefer the most recently indexed source
- Apply source authority scoring and conflict resolution rules (Correct answer)
- Randomly select one conflicting fact to retain
- Remove all facts involved in conflicts from the knowledge base
Correct answer: Apply source authority scoring and conflict resolution rules
Conflict resolution strategies assign trust scores to sources based on authority or provenance and apply rules (e.g., majority vote, recency weighting) to resolve contradictions.
Question 6: What is 'contextual compression' in RAG pipelines?
- Reducing the model's context window size to save memory
- Extracting only the relevant portions of retrieved documents before passing them to the LLM (Correct answer)
- Quantizing embeddings stored in the vector database
- Shortening queries to improve retrieval latency
Correct answer: Extracting only the relevant portions of retrieved documents before passing them to the LLM
Contextual compression post-processes retrieved documents to strip irrelevant content, passing only the most pertinent excerpts to the language model to reduce noise and token cost.
Question 7: In knowledge graph embedding models like TransE, how are relationships represented?
- As scalar weights between entity nodes
- As translations in vector space such that head + relation ≈ tail (Correct answer)
- As binary adjacency matrices between entity classes
- As attention weights computed over entity neighborhoods
Correct answer: As translations in vector space such that head + relation ≈ tail
TransE models relationships as vector translations, enforcing that the embedding of the head entity plus the relation vector approximates the embedding of the tail entity.
AI Knowledge Information 4
In information theory, what does 'entropy' measure in the context of a knowledge source?