Knowledge distillation and model extraction in AI

Knowledge distillation and model extraction in artificial intelligence can transfer selected behaviour or functionality from one model to another. They may reduce deployment costs and diffuse capabilities across firms or jurisdictions, but they are not the same technique and do not eliminate the need for compute, data, engineering or evaluation. The methods have no inherent state nexus and remain context unless a state directs, regulates or strategically exploits a specific campaign.

Distinct techniques

Knowledge distillation trains a student model to reproduce useful outputs or representations from a teacher, often to create a smaller or cheaper model. Synthetic-data training uses generated examples or labels, whether or not a student is smaller. Query-based model extraction uses repeated interactions to reproduce functionality, parameters or decision boundaries to a measured fidelity. Direct acquisition of a weights file is distinct from learning through an output interface, although some extraction methods can also recover model parameters.

Technical access, contractual permission, intellectual-property rights and export authorisation are separate questions. Authorised compression, open research and licence-compliant distillation should not be conflated with adversarial extraction or acquisition of protected data. Accuracy, fidelity and safety properties also require separate evaluation.

Cost and capability claims

Distillation reallocates compute rather than replacing it. The teacher must exist, outputs must be produced or collected, and the student still requires training, inference, infrastructure and testing. The strategic effect depends on the capability retained, total resources used, time, detectability and ability to deploy.

The DeepSeek-V3 technical report, Table 1 and its accompanying discussion, estimated 2.788 million H800 GPU hours and approximately US$5.576 million for the model's official training, assuming US$2 per GPU hour. That total combines pre-training, context extension and post-training; pre-training alone accounted for 2.664 million GPU hours and US$5.328 million. The authors excluded prior research and ablation experiments on architectures, algorithms or data. The reported training estimate is not a total company cost or an estimate for developing DeepSeek-R1.

DeepSeek's R1 report describes smaller models distilled from reported teacher outputs. Together with the developer's release statement permitting distillation, this documents a developer-reported example of capability transfer. It does not resolve every right or licence applicable to the base models. It does not prove that DeepSeek used any named rival model as the teacher for its frontier model. Such a claim would require evidence of the model, access path, outputs, method and legal basis.

Governance and statecraft relevance

Developers and cloud platforms can manage output access through identity checks, behavioural monitoring, contractual terms and API safeguards. Anthropic describes these measures in its February 2026 account; that account is evidence of the company's stated controls, not proof that they prevent every extraction attempt. Export rules can separately control specified chips, model weights, software, technology, end users and end uses. The controlled object and operative rule must be named rather than assuming that an export regime covers or excludes distillation as a category.

In May 2025 Commerce announced non-enforcement of the Artificial Intelligence Diffusion Rule and planned formal rescission. The Government Accountability Office's 12 May 2026 decision distinguished that enforcement policy from completed rulemaking. As at 18 October 2026, BIS's published EAR section 742.6 still contains the model-weights provisions referring to ECCN 4E091. The non-enforcement announcement does not mean all advanced-computing controls ceased: BIS's 31 May 2026 guidance expressly preserves pre-existing requirements for specified items supplied to entities headquartered in, or with ultimate parents headquartered in, Country Group D:5 or Macau, including in third countries. Its July 2026 UAE rule further distinguishes codified requirements from the destinations and recipients for which it enforces them. These provisions must be assessed by their defined items and parties; they do not establish a general prohibition on all model outputs. See United States AI diffusion rule, non-enforcement and destination-specific controls (2025-present).

Anthropic's February 2026 account of alleged coordinated extraction campaigns is a private company's detection finding. Its account and interaction figures require attribution and independent corroboration before being treated as established fact. A commercial or unauthorised campaign also does not become statecraft without evidence of state direction, sponsorship or strategic use.

See also

Artificial-intelligence export controls · United States AI diffusion rule, non-enforcement and destination-specific controls (2025-present) · Model weights as controlled technology · Hyperscale cloud infrastructure · Technology diffusion

Sources

1. Geoffrey Hinton, Oriol Vinyals and Jeff Dean, Distilling the Knowledge in a Neural Network (2015).
2. Cristian Buciluǎ, Rich Caruana and Alexandru Niculescu-Mizil, Model Compression, Proceedings of the Twelfth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2006).
3. Florian Tramèr, Fan Zhang, Ari Juels, Michael K. Reiter and Thomas Ristenpart, "Stealing Machine Learning Models via Prediction APIs", 25th USENIX Security Symposium (2016).
4. Matthew Jagielski, Nicholas Carlini, David Berthelot, Alex Kurakin and Nicolas Papernot, "High Accuracy and High Fidelity Extraction of Neural Networks", 29th USENIX Security Symposium (2020).
5. DeepSeek-AI, DeepSeek-V3 Technical Report (2024, revised 2025).
6. DeepSeek-AI, DeepSeek-R1: Incentivizing Reasoning Capability in Large Language Models via Reinforcement Learning (2025).
7. DeepSeek, "DeepSeek-R1 Release", 20 January 2025.
8. Anthropic, "Detecting and Preventing Distillation Attacks", 23 February 2026.
9. United States Department of Commerce, Bureau of Industry and Security, "Department of Commerce Announces Rescission of Biden-Era Artificial Intelligence Diffusion Rule and Strengthens Chip-Related Export Controls", 13 May 2025.
10. United States Department of Commerce, Bureau of Industry and Security, Export Administration Regulations, Part 744: Control Policy, End-User and End-Use Based, current edition, checked 29 July 2026.
11. United States Department of Commerce, Bureau of Industry and Security, "Department of Commerce Revises License Review Policy for Semiconductors Exported to China", 13 January 2026.
12. United States Government Accountability Office, Applicability of the Congressional Review Act to the Rescission of the Artificial Intelligence Diffusion Rule, B-337935, 12 May 2026.

1. United States Department of Commerce, Bureau of Industry and Security, Guidance Regarding Enforcement of License Requirements for Advanced Computing Items for Entities Headquartered in Country Group D:5 and Macau, 31 May 2026.
2. United States Department of Commerce, Bureau of Industry and Security, Enhanced Favorable Treatment for the United Arab Emirates Under the Export Administration Regulations, Federal Register 91, 43034-43039, 14 July 2026, effective 10 July 2026.
3. United States Department of Commerce, Bureau of Industry and Security, Export Administration Regulations, section 742.6, published regulatory text checked 18 October 2026.