{
 "question": "Context rot and knowledge decay controls for a long-running, AI-assisted archival project (Memory Commons, memorycommons.com \u2014 a deep-time human-knowledge preservation project built incrementally across many AI-agent sessions, with project state held in git repo docs, AI memory files, and published research). Research question: what is known about \"context rot\" and long-horizon knowledge degradation, and which concrete controls demonstrably prevent it? Sections: (A) LLM CONTEXT ROT (the AI-session failure mode): the Chroma Research \"Context Rot\" technical report (July 2025, 18 models incl. GPT-4.1, Claude 4, Gemini 2.5) \u2014 its actual findings on performance degradation with input length, needle-question similarity, distractors, haystack structure; related peer-reviewed literature: \"Lost in the Middle\" (Liu et al. 2023/TACL), RULER (NVIDIA), NoLiMa (2025), LongMemEval for long-term conversational memory; documented failure modes of agent memory across sessions: summarization/compaction information loss, stale memory reuse, fact drift \u2014 any published evaluations. (B) CONTEXT ENGINEERING CONTROLS for long-running agent projects: Anthropic's published context-engineering and agent-memory guidance (effective context engineering blog, memory tool docs), structured note-taking / external-memory patterns (files as memory, plan documents), just-in-time retrieval vs preloading, compaction best practice; industry-documented practices for multi-session AI coding projects (CLAUDE.md/AGENTS.md conventions, decision logs). (C) DOCUMENTATION & KNOWLEDGE DECAY in long-lived human projects (the decades failure mode): link rot rates from major studies (Harvard Law Review/Zittrain ~49-50% SCOTUS opinions, 404 rates over time, Pew 2024 \"When Online Content Disappears\" \u2014 38% of 2013 pages gone by 2023); documentation drift/staleness research in software engineering; Architecture Decision Records (ADR) practice \u2014 origin (Michael Nygard 2011), adoption evidence, benefits claims; docs-as-code and single-source-of-truth practices; knowledge loss when maintainers leave (bus factor studies). (D) ARCHIVAL SCIENCE ANTI-ROT CONTROLS (what preservation institutions standardized): OAIS reference model (ISO 14721) core concepts \u2014 SIP/AIP/DIP, fixity, preservation planning; PREMIS preservation metadata standard; NDSA Levels of Digital Preservation (the 4-level matrix: storage, integrity/fixity, control, metadata, content) \u2014 current version and what each level requires for fixity checking cadence; checksum audit practices at real archives (LOCKSS \"lots of copies keep stuff safe\" \u2014 model and evidence); format migration vs emulation debate outcomes. Output: cited, number-heavy findings per section with confidence levels; flag vendor claims vs peer-reviewed vs standards-body sources.",
 "summary": "All 25 claims refuted by adversarial verification. Research inconclusive \u2014 sources may be low-quality or claims overstated.",
 "findings": [],
 "refuted": [
  {
   "claim": "Chroma's Context Rot report (July 14, 2025) evaluated 18 LLMs \u2014 including Claude Opus 4, Claude Sonnet 4, GPT-4.1, o3, and Gemini 2.5 Pro/Flash \u2014 and found performance reliably degrades as input token count grows, even when the task itself is held constant.",
   "vote": "0-0",
   "source": "https://research.trychroma.com/context-rot"
  },
  {
   "claim": "The rate of degradation with input length depends on semantic similarity between the question and the target 'needle': lower needle-question similarity (measured ranges 0.445-0.775 for PG essays, 0.521-0.829 for arXiv) produces steeper performance decline as context grows.",
   "vote": "0-0",
   "source": "https://research.trychroma.com/context-rot"
  },
  {
   "claim": "Distractor content measurably harms retrieval: adding a single semantically-similar distractor lowers accuracy versus a distractor-free baseline, four distractors compound the effect, and distractor impact is non-uniform (specific distractors dominate hallucinated answers); Claude models showed the lowest hallucination rates and GPT models the highest.",
   "vote": "0-0",
   "source": "https://research.trychroma.com/context-rot"
  },
  {
   "claim": "Counterintuitively, models scored better when the haystack's sentences were shuffled than when the haystack kept its logical/coherent structure, indicating that how information is organized in context \u2014 not just whether it is present \u2014 drives long-context reliability.",
   "vote": "0-0",
   "source": "https://research.trychroma.com/context-rot"
  },
  {
   "claim": "On LongMemEval (long-term conversational memory), every model family scored significantly higher when given a focused prompt containing only the relevant parts than when given the full ~113k-token conversation history \u2014 direct evidence that just-in-time retrieval of minimal relevant context beats preloading full history.",
   "vote": "0-0",
   "source": "https://research.trychroma.com/context-rot"
  },
  {
   "claim": "Liu et al. (TACL 2024) found a U-shaped position effect: LLM performance on multi-document QA is highest when the relevant document is at the very beginning or end of the input context and degrades significantly when it is in the middle, even for explicitly long-context models.",
   "vote": "0-0",
   "source": "https://aclanthology.org/2024.tacl-1.9/"
  },
  {
   "claim": "GPT-3.5-Turbo's multi-document QA accuracy drops by more than 20 percentage points when relevant information sits mid-context; in the worst case (20- and 30-document settings) it falls below its closed-book accuracy of 56.1% (i.e., worse than having no documents at all).",
   "vote": "0-0",
   "source": "https://aclanthology.org/2024.tacl-1.9/"
  },
  {
   "claim": "Extended-context variants (GPT-3.5-Turbo 16K, Claude-1.3 100K) performed essentially identically to their base counterparts when input fit in both context windows, so larger context windows alone do not fix positional degradation.",
   "vote": "0-0",
   "source": "https://aclanthology.org/2024.tacl-1.9/"
  },
  {
   "claim": "NoLiMa evaluated 13 popular LLMs that claim to support contexts of at least 128K tokens, and found that at 32K tokens, 11 of the 13 models drop below 50% of their strong short-context baseline performance.",
   "vote": "0-0",
   "source": "https://arxiv.org/abs/2502.05167"
  },
  {
   "claim": "Even GPT-4o, one of the top-performing models on NoLiMa, degrades from a 99.3% short-context baseline to 69.7% as context length increases.",
   "vote": "0-0",
   "source": "https://arxiv.org/abs/2502.05167"
  },
  {
   "claim": "Standard needle-in-a-haystack benchmarks overstate long-context capability because models can exploit literal lexical matches between the needle and the question; NoLiMa removes this by designing needles with minimal lexical overlap, forcing latent-association inference.",
   "vote": "0-0",
   "source": "https://arxiv.org/abs/2502.05167"
  },
  {
   "claim": "Models that score near-perfectly on the simple needle-in-a-haystack test still show large performance degradation on RULER tasks as input length grows \u2014 i.e., vanilla NIAH overstates real long-context ability.",
   "vote": "0-0",
   "source": "https://github.com/NVIDIA/RULER"
  },
  {
   "claim": "Only about half of the evaluated models that claim a 32K+ context window can effectively handle 32K-token inputs, using Llama-2-7B's 4K performance (85.6%) as the qualitative threshold for 'effective' context length.",
   "vote": "0-0",
   "source": "https://github.com/NVIDIA/RULER"
  },
  {
   "claim": "Claimed context windows substantially exceed measured effective context: GPT-4-1106-preview claims 128K but is effective only to 64K (96.6% at 4K falling to 81.2% at 128K), and Llama3.1-70B claims 128K but is effective only to 64K (96.5% at 4K falling to 66.6% at 128K).",
   "vote": "0-0",
   "source": "https://github.com/NVIDIA/RULER"
  },
  {
   "claim": "Commercial chat assistants and long-context LLMs suffer roughly a 30% accuracy drop when required to memorize and use information across sustained multi-session interactions, as measured on the LongMemEval benchmark.",
   "vote": "0-0",
   "source": "https://arxiv.org/abs/2410.10813"
  },
  {
   "claim": "LongMemEval decomposes long-term conversational memory into five measurable abilities \u2014 information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention \u2014 giving a concrete taxonomy of long-horizon memory failure modes (including stale-knowledge/update failures relevant to fact drift).",
   "vote": "0-0",
   "source": "https://arxiv.org/abs/2410.10813"
  },
  {
   "claim": "The paper proposes concrete memory-system controls \u2014 session decomposition for value granularity, fact-augmented key expansion for indexing, and time-aware query expansion for retrieval \u2014 within a three-stage indexing/retrieval/reading framework, and reports these substantially improve memory recall and downstream QA (though the abstract gives no specific percentage gains).",
   "vote": "0-0",
   "source": "https://arxiv.org/abs/2410.10813"
  },
  {
   "claim": "Chroma's Context Rot report finds LLM performance is not uniform across context positions: performance varies significantly as input length grows, even on simple tasks, contradicting the assumption that the 10,000th token is handled as reliably as the 100th. (Vendor technical report, not peer-reviewed.)",
   "vote": "0-0",
   "source": "https://github.com/chroma-core/context-rot"
  },
  {
   "claim": "The repo is a full replication toolkit for the report, with code and datasets for three experiments: an extended Needle-in-a-Haystack testing semantic (non-lexical) needle-question matching and haystack-content variations, the LongMemEval benchmark, and a Repeated Words task \u2014 making the report's degradation findings independently reproducible.",
   "vote": "0-0",
   "source": "https://github.com/chroma-core/context-rot"
  },
  {
   "claim": "On the LoCoMo benchmark for very long-term conversational memory, even the best-performing configurations using long-context LLMs or RAG remain far below human performance: gpt-3.5-turbo-16k scored 37.8 overall F1 versus a human benchmark of 87.9 (Table 2), a ~50-point gap.",
   "vote": "0-0",
   "source": "https://arxiv.org/abs/2402.17753"
  },
  {
   "claim": "Temporal reasoning is the most degraded capability in long-horizon conversations: gpt-3.5-turbo-16k scored only 20.3 F1 on temporal-reasoning questions versus 92.6 for humans (Table 2), i.e. models fail to track when things happened across sessions.",
   "vote": "0-0",
   "source": "https://arxiv.org/abs/2402.17753"
  },
  {
   "claim": "Long-context models are highly prone to asserting unsupported information over long dialogues: the top long-context model's performance on adversarial (unanswerable) questions collapsed to 2.1%, evidence of hallucination/fact-drift rather than abstention when memory is uncertain.",
   "vote": "0-0",
   "source": "https://arxiv.org/abs/2402.17753"
  },
  {
   "claim": "Across seven model families (DeepSeek-V4-Flash, GLM-5.1, Qwen3.6-27B, Kimi-K2.5, Claude-Sonnet-4.6, GPT-5.4-mini, Gemini-3.5-flash; 1,323 episodes), context compaction raised the rate at which agents violated an in-context governance constraint from 0% (policy in full context) to a pooled 30%, up to 59% on DeepSeek-V4 and Kimi-K2.5; violation was 0% when the constraint survived the summary and 38% when it was dropped.",
   "vote": "0-0",
   "source": "https://arxiv.org/pdf/2606.22528"
  },
  {
   "claim": "Ablations show the summarizer/compactor, not generic long-context degradation, causes the failure: GLM-5.1 never violates in an uncompressed 5.9k-token context (0%), but a counterfactual summary omitting the policy yields 60% violation, and restoring the policy to the same summary returns violation to 0%; in cross-model panels violation tracks the summarizer, not the agent.",
   "vote": "0-0",
   "source": "https://arxiv.org/pdf/2606.22528"
  },
  {
   "claim": "Constraint Pinning \u2014 quarantining governance constraints from lossy compaction and re-injecting them verbatim after every compaction step with integrity checks \u2014 is a training-free control that restored violation to 0% across all seven models and both fixed attack variants, at a cost of about 47 pinned tokens (<0.5% of production-scale compaction contexts), while completing 99% of allowed actions (1% over-refusal).",
   "vote": "0-0",
   "source": "https://arxiv.org/pdf/2606.22528"
  }
 ],
 "sources": [
  {
   "url": "https://research.trychroma.com/context-rot",
   "quality": "primary",
   "claimCount": 5
  },
  {
   "url": "https://aclanthology.org/2024.tacl-1.9/",
   "quality": "primary",
   "claimCount": 5
  },
  {
   "url": "https://arxiv.org/abs/2502.05167",
   "quality": "primary",
   "claimCount": 5
  },
  {
   "url": "https://github.com/NVIDIA/RULER",
   "quality": "primary",
   "claimCount": 5
  },
  {
   "url": "https://arxiv.org/abs/2410.10813",
   "quality": "primary",
   "claimCount": 5
  },
  {
   "url": "https://github.com/chroma-core/context-rot",
   "quality": "primary",
   "claimCount": 5
  },
  {
   "url": "https://arxiv.org/abs/2402.17753",
   "quality": "primary",
   "claimCount": 5
  },
  {
   "url": "https://arxiv.org/pdf/2606.22528",
   "quality": "primary",
   "claimCount": 5
  },
  {
   "url": "https://arxiv.org/html/2605.12493v1",
   "quality": "primary",
   "claimCount": 5
  },
  {
   "url": "https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents",
   "quality": "primary",
   "claimCount": 5
  },
  {
   "url": "https://platform.claude.com/docs/en/agents-and-tools/tool-use/memory-tool",
   "quality": "primary",
   "claimCount": 5
  },
  {
   "url": "https://platform.claude.com/cookbook/tool-use-context-engineering-context-engineering-tools",
   "quality": "primary",
   "claimCount": 5
  },
  {
   "url": "https://code.claude.com/docs/en/best-practices",
   "quality": "primary",
   "claimCount": 5
  },
  {
   "url": "https://code.claude.com/docs/en/memory",
   "quality": "primary",
   "claimCount": 5
  },
  {
   "url": "https://redreamality.com/blog/claude-md-agents-md-deep-dive/",
   "quality": "blog",
   "claimCount": 5
  },
  {
   "url": "https://www.ndsa.org/2026/03/23/announcing-version-2_1-of-the-ndsa-levels-of-digital-preservation.html",
   "quality": "primary",
   "claimCount": 5
  },
  {
   "url": "https://www.digitalpreservation.gov/documents/NDSA-Fixity-Guidance-Report-final100214.pdf",
   "quality": "primary",
   "claimCount": 5
  }
 ],
 "stats": {
  "angles": 5,
  "sources": 17,
  "claims": 85,
  "verified": 25,
  "confirmed": 0,
  "killed": 25
 }
}