Streaming 與用戶端範例

SSE 格式

data: {"token": "Hello", "finish_reason": null}
data: {"token": " world", "finish_reason": null}
data: {"token": "", "finish_reason": "stop"}
data: [DONE]

curl 範例

curl -N http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"tinyloop","messages":[{"role":"user","content":"Hi"}],"stream":true}'

Python 範例

import requests
r = requests.post("http://localhost:8000/generate",
                   json={"prompt": "Hello", "max_tokens": 32})
print(r.json()["text"])

Smoke test

cd server && python smoke_test.py