diff --git a/docs/config-local-oobabooga.md b/docs/config-local-oobabooga.md index d27996dbf..c8f415d2e 100644 --- a/docs/config-local-oobabooga.md +++ b/docs/config-local-oobabooga.md @@ -4,7 +4,7 @@ Integrate local Large Language Models (LLMs) with [oobabooga/text-generation-webui](https://github.com/oobabooga/text-generation-webui), a specialized interface that includes a custom variant of the OpenAI API for a smooth integration process. -_Last updated on Nov 7, 2023_ +_Last updated on Dec 7, 2023_ ### Components @@ -20,26 +20,35 @@ This guide assumes that **big-AGI** is already installed on your system. Note th ### Text-web-ui Installation & Configuration: -1. Install [text-generation-webui](https://github.com/oobabooga/text-generation-webui#Installation). - - Download the one-click installer, extract it, and double-click on "start" - ~10 minutes - - Close it afterwards as we need to modify the startup flags +1. Install [text-generation-webui](https://github.com/oobabooga/text-generation-webui#Installation): + - Follow the instructions in the official page (basicall clone the repo and run a script) [~10 minutes] + - Stop the Web UI as we need to modify the startup flags to enable the OpenAI API 2. Enable the **openai extension** - Edit `CMD_FLAGS.txt` - - Make sure that `--listen --extensions openai` is present and uncommented + - Make sure that `--listen --api` is present and uncommented 3. Restart text-generation-webui - Double-click on "start" - You should see something like: ``` - 2023-11-07 21:24:26 INFO:Loading the extension "openai"... - 2023-11-07 21:24:27 INFO:OpenAI compatible API URL: + 2023-12-07 21:51:21 INFO:Loading the extension "openai"... + 2023-12-07 21:51:21 INFO:OpenAI-compatible API URL: - http://0.0.0.0:5000/v1 + http://0.0.0.0:5000 + + 2023-12-07 21:51:21 INFO:Loading the extension "gallery"... + INFO: Started server process [24756] + INFO: Waiting for application startup. + INFO: Application startup complete. + INFO: Uvicorn running on http://0.0.0.0:5000 (Press CTRL+C to quit) + Running on local URL: http://0.0.0.0:7860 ``` - - The OpenAI API is now running on port 5000, on both localhost (127.0.0.1) and your network IP address + - This shows that: + - The Web UI is running on port 7860: http://127.0.0.1:7860 + - **The OpenAI API is running on port 5000: http://127.0.0.1:5000** 4. Load your first model - Open the text-generation-webui at [127.0.0.1:7860](http://127.0.0.1:7860/) - Switch to the **Model** tab - - Download, for instance, `TheBloke/Llama-2-7b-Chat-GPTQ:gptq-4bit-32g-actorder_True` - 4.3 GB + - Download, for instance, `TheBloke/Llama-2-7B-Chat-GPTQ` - Select the model once it's loaded ### Integrating text-web-ui with big-AGI: @@ -51,4 +60,6 @@ This guide assumes that **big-AGI** is already installed on your system. Note th - The active model must be selected and LOADED on the text-generation-webui as it doesn't support model switching or parallel requests. - Select model & Chat +![config-oobabooga-0.png](pixels/config-oobabooga-0.png) + Enjoy the privacy and flexibility of local LLMs with `big-AGI` and `text-generation-webui`! \ No newline at end of file diff --git a/docs/pixels/config-oobabooga-0.png b/docs/pixels/config-oobabooga-0.png new file mode 100644 index 000000000..06c800e72 Binary files /dev/null and b/docs/pixels/config-oobabooga-0.png differ