zkLLM: Zero Knowledge Proofs for Large Language Models
发表信息
作者
- Haochen Sun
- Jason Li
- Hongyang Zhang
笔记
The recent surge in artificial intelligence (AI), characterized by the prominence of large language models (LLMs), has ushered in fundamental transformations across the globe. However, alongside these advancements, concerns surrounding the legitimacy of LLMs have grown, posing legal challenges to their extensive applications. Compounding these concerns, the parameters of LLMs are often treated as intellectual property, restricting direct investigations. In this study, we address a fundamental challenge within the realm of AI legislation: the need to establish the authenticity of outputs generated by LLMs. To tackle this issue, we present zkLLM, which stands as the inaugural specialized zero-knowledge proof tailored for LLMs to the best of our knowledge. Addressing the persistent challenge of non-arithmetic operations in deep learning, we introduce tlookup, a parallelized lookup argument designed for non-arithmetic tensor operations in deep learning, offering a solution with no asymptotic overhead. Furthermore, leveraging the foundation of tlookup, we introduce zkAttn, a specialized zero-knowledge proof crafted for the attention mechanism, carefully balancing considerations of running time, memory usage, and accuracy. Empowered by our fully parallelized CUDA implementation, zkLLM emerges as a significant stride towards achieving efficient zero-knowledge verifiable computations over LLMs. Remarkably, for LLMs boasting 13 billion parameters, our approach enables the generation of a correctness proof for the entire inference process in under 15 minutes. The resulting proof, compactly sized at less than 200 kB, is designed to uphold the privacy of the model parameters, ensuring no inadvertent information leakage.
近年来，以大语言模型(large language models, LLMs)为代表的人工智能(artificial intelligence, AI)的迅猛发展，在全球范围内引发了根本性的变革。然而，伴随这些进步，围绕LLMs合法性的担忧也日益增长，对其广泛应用构成了法律挑战。更为复杂的是，LLMs的参数通常被视为知识产权，限制了直接研究的可能性。
在本研究中，我们着眼于AI立法领域的一个基本挑战：建立LLMs生成输出的真实性验证机制。为解决这一问题，我们提出了zkLLM，据我们所知，这是首个专门针对LLMs的零知识证明系统。为应对深度学习中非算术运算这一持续存在的挑战，我们引入了tlookup，这是一种针对深度学习中非算术张量运算的并行化查找论证方法，提供了一个没有渐近开销的解决方案。此外，基于tlookup的基础，我们提出了zkAttn，这是一种专门针对注意力机制的零知识证明，在运行时间、内存使用和准确性之间实现了精心的平衡。
通过我们完全并行化的CUDA实现，zkLLM在实现LLMs高效零知识可验证计算方面取得了重大进展。值得注意的是，对于拥有130亿参数的LLMs，我们的方法能够在15分钟内为整个推理过程生成正确性证明。生成的证明大小紧凑，不到200 kB，同时设计上确保了模型参数的隐私性，防止任何无意的信息泄露。