Skip to main content
Llamafile 让您可以通过单个文件分发和运行 LLM。 Llamafile 将 llama.cpp 与 Cosmopolitan Libc 整合到一个框架中，将 LLM 的所有复杂性压缩为单一可执行文件（称为”llamafile”），可在大多数计算机上本地运行，无需安装。

​
设置

  1. 下载您想使用的模型对应的 llamafile。您可以在 HuggingFace 上找到许多 llamafile 格式的模型。本指南中，我们将下载一个较小的模型 TinyLlama-1.1B-Chat-v1.0.Q5_K_M。注意：如果您没有安装 wget，可以直接通过此链接下载该模型。
wget https://huggingface.co/jartine/TinyLlama-1.1B-Chat-v1.0-GGUF/resolve/main/TinyLlama-1.1B-Chat-v1.0.Q5_K_M.llamafile
  1. 使 llamafile 可执行。首先，如果您尚未打开终端，请先打开。如果您使用 macOS、Linux 或 BSD， 需要使用 chmod 授权运行此新文件（见下方）。如果您使用 Windows， 请在文件名末尾添加 “.exe” 重命名该文件（模型文件应命名为 TinyLlama-1.1B-Chat-v1.0.Q5_K_M.llamafile.exe）。
chmod +x TinyLlama-1.1B-Chat-v1.0.Q5_K_M.llamafile  # run if you're on MacOS, Linux, or BSD
  1. 以”服务器模式”运行 llamafile：
./TinyLlama-1.1B-Chat-v1.0.Q5_K_M.llamafile --server --nobrowser
现在您可以调用 llamafile 的 REST API。默认情况下，llamafile 服务器监听 localhost:8080。您可以在此处查看完整的服务器文档。您可以直接通过 REST API 与 llamafile 交互，但这里我们将展示如何使用 LangChain 与其交互。

​
使用方法

from langchain_community.llms.llamafile import Llamafile

llm = Llamafile()

llm.invoke("Tell me a joke")

'? \nI\'ve got a thing for pink, but you know that.\n"Can we not talk about work anymore?" - What did she say?\nI don\'t want to be a burden on you.\nIt\'s hard to keep a good thing going.\nYou can\'t tell me what I want, I have a life too!'
要流式传输 token，请使用 .stream(...) 方法： 
query = "Tell me a joke"

for chunks in llm.stream(query):
    print(chunks, end="")

print()

.
- She said, "I'm tired of my life. What should I do?"
- The man replied, "I hear you. But don't worry. Life is just like a joke. It has its funny parts too."
- The woman looked at him, amazed and happy to hear his wise words. - "Thank you for your wisdom," she said, smiling. - He replied, "Any time. But it doesn't come easy. You have to laugh and keep moving forward in life."
- She nodded, thanking him again. - The man smiled wryly. "Life can be tough. Sometimes it seems like you're never going to get out of your situation."
- He said, "I know that. But the key is not giving up. Life has many ups and downs, but in the end, it will turn out okay."
- The woman's eyes softened. "Thank you for your advice. It's so important to keep moving forward in life," she said. - He nodded once again. "You're welcome. I hope your journey is filled with laughter and joy."
- They both smiled and left the bar, ready to embark on their respective adventures.
Edit this page on GitHub or file an issue.
Connect these docs to Claude, VSCode, and more via MCP for real-time answers.