mirror of
https://github.com/enricoros/big-AGI.git
synced 2026-05-10 21:50:14 -07:00
Oobabooga: document the changes
This commit is contained in:
@@ -4,7 +4,7 @@ Integrate local Large Language Models (LLMs) with
|
||||
[oobabooga/text-generation-webui](https://github.com/oobabooga/text-generation-webui),
|
||||
a specialized interface that includes a custom variant of the OpenAI API for a smooth integration process.
|
||||
|
||||
_Last updated on Nov 7, 2023_
|
||||
_Last updated on Dec 7, 2023_
|
||||
|
||||
### Components
|
||||
|
||||
@@ -20,26 +20,35 @@ This guide assumes that **big-AGI** is already installed on your system. Note th
|
||||
|
||||
### Text-web-ui Installation & Configuration:
|
||||
|
||||
1. Install [text-generation-webui](https://github.com/oobabooga/text-generation-webui#Installation).
|
||||
- Download the one-click installer, extract it, and double-click on "start" - ~10 minutes
|
||||
- Close it afterwards as we need to modify the startup flags
|
||||
1. Install [text-generation-webui](https://github.com/oobabooga/text-generation-webui#Installation):
|
||||
- Follow the instructions in the official page (basicall clone the repo and run a script) [~10 minutes]
|
||||
- Stop the Web UI as we need to modify the startup flags to enable the OpenAI API
|
||||
2. Enable the **openai extension**
|
||||
- Edit `CMD_FLAGS.txt`
|
||||
- Make sure that `--listen --extensions openai` is present and uncommented
|
||||
- Make sure that `--listen --api` is present and uncommented
|
||||
3. Restart text-generation-webui
|
||||
- Double-click on "start"
|
||||
- You should see something like:
|
||||
```
|
||||
2023-11-07 21:24:26 INFO:Loading the extension "openai"...
|
||||
2023-11-07 21:24:27 INFO:OpenAI compatible API URL:
|
||||
2023-12-07 21:51:21 INFO:Loading the extension "openai"...
|
||||
2023-12-07 21:51:21 INFO:OpenAI-compatible API URL:
|
||||
|
||||
http://0.0.0.0:5000/v1
|
||||
http://0.0.0.0:5000
|
||||
|
||||
2023-12-07 21:51:21 INFO:Loading the extension "gallery"...
|
||||
INFO: Started server process [24756]
|
||||
INFO: Waiting for application startup.
|
||||
INFO: Application startup complete.
|
||||
INFO: Uvicorn running on http://0.0.0.0:5000 (Press CTRL+C to quit)
|
||||
Running on local URL: http://0.0.0.0:7860
|
||||
```
|
||||
- The OpenAI API is now running on port 5000, on both localhost (127.0.0.1) and your network IP address
|
||||
- This shows that:
|
||||
- The Web UI is running on port 7860: http://127.0.0.1:7860
|
||||
- **The OpenAI API is running on port 5000: http://127.0.0.1:5000**
|
||||
4. Load your first model
|
||||
- Open the text-generation-webui at [127.0.0.1:7860](http://127.0.0.1:7860/)
|
||||
- Switch to the **Model** tab
|
||||
- Download, for instance, `TheBloke/Llama-2-7b-Chat-GPTQ:gptq-4bit-32g-actorder_True` - 4.3 GB
|
||||
- Download, for instance, `TheBloke/Llama-2-7B-Chat-GPTQ`
|
||||
- Select the model once it's loaded
|
||||
|
||||
### Integrating text-web-ui with big-AGI:
|
||||
@@ -51,4 +60,6 @@ This guide assumes that **big-AGI** is already installed on your system. Note th
|
||||
- The active model must be selected and LOADED on the text-generation-webui as it doesn't support model switching or parallel requests.
|
||||
- Select model & Chat
|
||||
|
||||

|
||||
|
||||
Enjoy the privacy and flexibility of local LLMs with `big-AGI` and `text-generation-webui`!
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 730 KiB |
Reference in New Issue
Block a user