Unredacted Filings Expose Foundation Model Ingestion Tactics and Threaten Fair Use Defenses

Newly unsealed filings in The New York Times copyright lawsuit against OpenAI and Microsoft have surfaced internal communications that directly challenge the foundational legal defenses supporting commercial large language models (LLMs). According to reporting by TechCrunch on the unredacted brief, executives and researchers across both companies privately acknowledged that conversational AI systems act as direct market substitutes for digital publishers, circumvented access controls, and systematically scrubbed attribution metadata from training corpuses.

While the excerpts originate from the plaintiff's brief and the underlying source exhibits remain sealed, the documented statements and internal metrics create significant operational and legal exposure for AI labs relying on broad web scrapes under the umbrella of copyright fair use.

The Fair Use Vulnerability: Market Substitution and Economic Impact

For years, frontier AI developers have defended unlicensed pre-training by invoking the fair use doctrine, arguing that model weights transform source text into mathematical representations rather than distributing copies. However, a central factor in four-part fair use evaluations is the effect of the use upon the potential market for or value of the copyrighted work.

The unredacted filings introduce internal corporate findings that contradict the argument that conversational inference does not cannibalize web traffic:

• Quantified Referral Collapse: Microsoft's internal metrics revealed that its Copilot "answer engine" caused user click-through rates (CTR) for The New York Times domain to plummet by as much as 93% compared to standard Bing search queries.
• Executive Concessions on Substitution: Nick Turley, OpenAI's Head of ChatGPT, acknowledged in internal correspondence that the chatbot presents an "existential threat" to publishers, characterizing the product as "largely substitutive" and projecting that it "will get more and more substitutive as they get better."
• Deposition Statements: Under oath, Microsoft CEO Satya Nadella testified that conversational AI "has substituted ... giving you the information right there on the website on the AI platform versus needing to go to the underlying source."

These admissions undermine the argument that AI-generated summaries serve merely as transformative pointers to original reporting. When an answer engine directly satisfies user queries using ingested journalistic analysis, courts are far more likely to classify the system as an unlicensed market replacement.

The Content Pipeline: Scrapes, Shared Datasets, and Paywall Circumvention

Beyond market substitution, the unredacted filings detail the specific engineering practices deployed by both organizations to compile training corpuses at scale. The records outline how datasets such as WebText, WebText2, Common Crawl derivatives, and proprietary indexes were assembled.

Scale of Ingested Works

The brief details that OpenAI's mid-training datasets contained more than 91,692 copies of works published by the Times, the Daily News, and the Center for Investigative Reporting. A single Common Crawl-derived dataset contained more than 2 million documents from nytimes.com alone.

Cross-Company Ingestion Pipelines

The filing outlines technical collaboration initiatives between the two tech giants:

• Dataset Sharing: OpenAI provided its complete GPT-3 training dataset to Microsoft to assist the company in evaluating commercial product integrations.
• Project Taxi and Project Mango: Microsoft supplied training data back to OpenAI through internal initiatives dubbed Project Taxi and Project Mango. Data assembled via Project Mango alone incorporated at least 160,903 unique works sourced from the plaintiff news organizations, leveraging scraping pipelines tied to the Bing search index.

Deliberate Evasion and Metadata Stripping

Two procedural disclosures in the brief present distinct legal liabilities under the Digital Millennium Copyright Act (DMCA):

1. Paywall Bypass Mechanisms: Communications show OpenAI researcher Nick Ryder informing OpenAI President Greg Brockman of a functional "hack to get around nytimes paywall," to which Brockman acknowledged, "ah nice." The intentional evasion of technological access controls directly weakens defenses claiming that scraping was confined to freely accessible public web data.
2. Copyright Management Information (CMI) Removal: The filing alleges deliberate filtering routines designed to strip copyright notices and author attributions from training pipelines. Internal notes indicated that engineers removed these identifiers specifically because researchers "wouldn't want model outputting copyright notices" to end users.

The 'Doom Loop' and Content Supply Chain Fragility

The legal friction highlights a structural tension within AI infrastructure: foundation models depend on high-quality human-generated data, yet deploying those models undermines the financial viability of primary content creators.

In a January 2024 internal presentation, Microsoft Director of Applied Science Brent Hecht termed this feedback cycle a "doom loop" that threatened to "hurt the performance of our models and the entire web at the same time." Hecht noted: "It is highly unusual that an end-product threatens the economic foundations of its essential suppliers, but that is the situation we have created for our LLM business with respect to its 'content supply chain.'" A year prior, in a January 2023 memo, Hecht went further, describing uncompensated web scraping as "the largest theft of labor in human history."

This internal friction mirrors executive testimony regarding licensing boundaries. In his deposition, Nadella stated that "anything that is paywalled should be licensed by anyone who wants to use it... for grounding or training," stating that if he had known OpenAI scraped paywalled content, he would have exercised Microsoft's rights to require OpenAI to retrain its models.

Strategic Consequences for Model Builders and Data Engineers

As this litigation advances, the evidentiary disclosures create practical ramifications for enterprise AI teams and machine learning infrastructure developers:

• Auditing Data Provenance: Relying on unvetted public corpuses or third-party web dumps poses escalating legal and operational risks. Enterprise teams must implement verifiable lineage tracking for all pre-training, fine-tuning, and retrieval-augmented generation (RAG) datasets.
• Accelerated Shift to Paid Licensing: Bilateral data licensing agreements—already established by OpenAI with several media conglomerates—are shifting from an optional public relations gesture to a mandatory operational requirement for training frontier models.
• Evolving RAG and Retrieval Architectures: For production RAG systems and answer engines, systems that bypass publisher access controls or fail to preserve CMI and referral value will face increased scrutiny from rights holders deploying technical blocks, legal challenges, and paywall hardening.
• Focus on Synthetic and Curated Data: As web scraping faces legal limits, research labs will likely intensify engineering investments into high-fidelity synthetic data generation and specialized, fully licensed domain-specific corpuses.

The unsealed materials demonstrate that internal technical teams were acutely aware of the economic and structural disruption caused by foundation model training pipelines. As courts evaluate whether fair use protects the extraction of value from closed and commercial web properties, AI developers must prepare for a future defined by rigorous data accounting, strict licensing boundaries, and significant compliance overhead.