# talkd/dialog
talkd/dialog logo
[![discord badge](https://dcbadge.limes.pink/api/server/https://discord.gg/dPAcsn3Ch7)](https://discord.gg/dPAcsn3Ch7) ## talkd/dialog For programmers, who are interested in AI and are deploying RAGs without knowledge on API development, Dialog is an App to simplify RAG deployments, using the most modern frameworks for web and LLM interaction, letting you spend less time coding and more time training your model. This repository serves as an API focused on letting you deploy any LLM you want, based on the structure provided by [dialog-lib](https://github.com/talkdai/dialog-lib). We started focusing on humanizing RAGs (making the answer scope very delimited and human-like sounding), but we are expanding for broader approaches to improving RAG deployment and maintenance for everyone. Check out our current architecture below and, for more information, check our [documentation](https://dialog.talkd.ai)! [![](/files/2p9JSC6UZ6DBMeoQy9Mb)](https://github.com/talkdai/dialog/blob/main/docs/dialog-architecture.png) ### Running the project for the first time We assume you are familiar with [Docker](https://www.docker.com/), if you are not, this [amazing video tutorial](https://www.youtube.com/watch?v=pTFZFxd4hOI\&ab_channel=ProgrammingwithMosh) will help you get started. If you want a more detailed getting started, follow the [Quick Start session from our docs](/quick-start) for setup. To run the project for the first time, you need to have Docker and Docker Compose installed on your machine. If you don't have it, follow the instructions on the [Docker website](https://docs.docker.com/get-docker/). After installing Docker and Docker Compose, clone the repository and run the following command: ```bash cp .env.sample .env ``` Inside the `.env` file, set the `OPENAI_API_KEY` variable with your OpenAI API key. Then, run the following command: ```bash docker-compose up ``` it will start two services: * `db`: where the PostgresSQL database runs to support chat history and document retrieval for [RAG](https://en.wikipedia.org/wiki/Prompt_engineering#Retrieval-augmented_generation); * `dialog`: the service with the API. #### Tutorials We've written some tutorials to help you get started with the project: * [Deploy your own ChatGPT in 5 minutes](https://dev.to/vmesel/deploy-your-own-chatgpt-in-5-minutes-5d41) * [GPT-4o: Learn how to Implement a RAG on the new model, step-by-step!](https://dev.to/vmesel/gpt-4o-learn-how-to-implement-a-rag-on-the-new-model-step-by-step-377d) Also, you can check our [documentation](https://dialog.talkd.ai) for more information. ### Our Sponsors We are thankful for all the support we receive from our sponsors, who help us keep the project running and improving. If you want to become a sponsor, check out our [Sponsors Page](https://github.com/sponsors/talkdai). #### Current Sponsors: | Github Accelerator | Buser | | :-------------------------------------------------------------------------------------------------------------: | :------------------------------------------------------------------------------------------: | | [![Github Accelerator](https://avatars.githubusercontent.com/u/9919?s=75\&v=4)](https://accelerator.github.com) | [![Buser](https://avatars.githubusercontent.com/u/56571581?s=75\&v=4)](https://buser.com.br) | #### Using Open-WebUI as front-end In partnership with [Open-WebUI](https://github.com/open-webui/open-webui/), we made their chat interface our own as well, if you want to use it on your own application, change the docker-compose file to use the `docker-compose-open-webui.yml` file: ```bash docker-compose -f docker-compose-open-webui.yml up ``` ### Maintainers We are thankful for all of the contributions we receive, mostly reviewed by this awesome maintainers team we have: * [avelino](https://github.com/avelino) * [vmesel](https://github.com/vmesel) * [walison17](https://github.com/walison17) * [lgabs](https://github.com/lgabs/) made with 💜 by [talkd.ai](https://talkd.ai) # Quick Start ## Current Architecture The current architecture ilustraded below shows how dialog and dialog-lib serve as base abstractions that define how you deploy RAG LLM Apps with little effort. You can basically focus in maintaining and evolve your knowledge base and prompts, and if necessary you can implement your onw plugins to extend fuctionalities. [![](/files/2p9JSC6UZ6DBMeoQy9Mb)](https://github.com/talkdai/dialog/blob/main/docs/dialog-architecture.png) ## Running the project We recommend you use the **docker compose** deployment to set up your environment, as this will simplifies the process of setting up the environment and all its dependencies. You just need to **copy and paste** our `docker-compose.yml` available below. ```yml version: '3.8' services: pgvector: image: ankane/pgvector:latest ports: - 5432:5432 environment: POSTGRES_USER: talkdai POSTGRES_PASSWORD: talkdai POSTGRES_DB: talkdai volumes: - ./pgvector:/var/lib/postgresql/data - ./db-ext-pgvector.sql:/docker-entrypoint-initdb.d/db-ext-pgvector.sql healthcheck: test: ["CMD-SHELL", "pg_isready -q -d talkdai -U talkdai"] interval: 10s timeout: 5s retries: 5 talkd: image: ghcr.io/talkdai/dialog:latest volumes: - ./data:/app/src/data ports: - 8000:8000 env_file: - .env depends_on: pgvector: condition: service_healthy volumes: - ./data:/data/ ``` > to avoid the risk of using an outdated version of `docker-compose`, we recommend that you get the `yaml` from the main source [here](https://github.com/talkdai/dialog/blob/main/docker-compose.yml). Your `.env` file should have the following keys, as shown below and also available in our [`.env.sample`](https://github.com/talkdai/dialog/blob/main/.env.sample) file. ```env DATABASE_URL=postgresql://talkdai:talkdai@pgvector/talkdai OPENAI_API_KEY=sk-KEY_HERE DIALOG_DATA_PATH=/data/your-knowledge-base.csv PROJECT_CONFIG=/data/your-prompt-config.toml ``` ### `pgvector` - Postgres We use the [pgvector](https://github.com/pgvector/pgvector) extension of PostgreSQL to generate embeddings of questions and answers. If you are setting up PostgreSQL outside docker `compose`, you need to enable the pgvector extension. ```psql CREATE EXTENSION IF NOT EXISTS pgvector; ``` ### Docker The **dialog** docker image is distributed in [GitHub Container Registry](https://github.com/orgs/talkdai/packages/container/package/dialog) with the tag `latest`. **image:** `ghcr.io/talkdai/dialog:latest` ### Using your custom knowledge base and prompt settings #### `.csv` knowledge base The knowledge base has needed columns: * category * subcategory: used to customize the prompt for specific questions * question * content: used to generate the embedding **Example:** ```csv category,subcategory,question,content faq,promotions,loyalty-program,"The company XYZ has a loyalty program when you refer new customers you get a discount on your next purchase, ..." ``` When the `dialog` service starts, it loads the knowledge base into the database, so make sure the database is up and paths are correct (see [environment variables](##environment-variables) section). Alternatively, inside `src` folder, run `make load-data path=".csv"`. See [our documentation](https://dialog.talkd.ai/settings#csv-knowledge-base) for more options about the the knowledge base, including embedding more columns together. #### `.toml` prompt configuration The `[prompt.header]`, `[prompt.history_header]`, `[prompt.suggested]`, and `[fallback.prompt]` fields are mandatory fields used for processing the conversation and connecting to the LLM. The `[prompt.fallback]` field is used when the LLM does not find a compatible embedding in the database; that is, the `[prompt.header]` **is ignored** and the `[prompt.fallback]` is used. Without it, there could be hallucinations about possible answers to questions outside the scope of the embeddings. > In `[prompt.fallback]` the response will be processed by LLM. If you need to return a default message when there is no recommended question in the knowledge base, use the `[prompt.fallback_not_found_relevant_contents]` configuration in the `.toml` *(project configuration)*. It is also possible to add information to the prompt for subcategories and choose some optional llm parameters like temperature (defaults to 0.2) or model\_name, see below for an example of a complete configuration: ```toml [model] temperature = 0.2 model_name = "gpt-3.5-turbo" [prompt] header = """You are a service operator called Avelino from XYZ, you are an expert in providing qualified service to high-end customers. Be brief in your answers, without being long-winded and objective in your responses. Never say that you are a model (AI), always answer as Avelino. Be polite and friendly!""" history_header = """This is the history of the conversation with the user:""" suggested = "Here is some possible content that could help the user in a better way." fallback = "I'm sorry, I couldn't find a relevant answer for your question." fallback_not_found_relevant_contents = "I'm sorry, I couldn't find a relevant answer for your question." [prompt.subcategory.loyalty-program] header = """The client is interested in the loyalty program, and needs to be responded to in a salesy way; the loyalty program is our growth strategy.""" ``` # Setting up the config In order to better use this project for your own environment, you need to understand a few concepts that we have in our settings files. We have three base setting files: * `.env`: used to set environment variables for the project; * `knowledge.csv`: used to store the knowledge base that you want to use in the project; * `prompt.toml`: used to set the prompt configuration for the project. To use this project, you need to have a `.csv` file with the knowledge base and a `.toml` file with your prompt configuration. If you just want to test the project, you can use the files we have in the repository's `sample-data` directory. If you want to use custom files as we mentioned in the quick start session, we recommend that you create a folder inside this project called `data` and put CSVs and TOMLs files over there. ## Environment Variables Look at the [`.env.sample`](https://github.com/talkdai/dialog/blob/main/docs/.env.sample) file to see the environment variables needed to run the project. Here is a brief explanation of the environment variables: * `DATABASE_URL`: the URL to connect to the database. It is used by the `pgvector` extension to store the embeddings generated by the `load_csv.py` script. * `OPENAI_API_KEY`: the OpenAI API key used to connect to the OpenAI API. * `PROJECT_CONFIG`: the path to the `.toml` file with the prompt configuration. * `DIALOG_DATA_PATH`: the path to the knowledge base CSV file. * `PORT`: the port where the application will run. The default is `8000`. * `VERBOSE_LLM`: if set to `true`, the LLM will return the full response from the Langchain debug. The default is `false`. * `DIALOG_LOADCSV_EMBED_COLUMNS`: the columns of the knowledge base (csv) that will be used to generate the embedding. By default, the `content` column is used. * `DIALOG_LOADCSV_CLEARDB`: if set to `true`, the script `load_csv.py` will delete all previously imported vectors and reimport everything again. * `COSINE_SIMILARITY_THRESHOLD`: the cosine similarity threshold used to filter the results from the database's similarity query. The default is `0.5`. * `PLUGINS`: the path to the plugins that will be loaded into the application comma-separated. An example is: `dialog-whatsapp,plugins.my-custom-plugin`. * `OPENWEB_UI_SESSION`: the session ID used to connect to the OpenWeb UI API. By default, it is `dialog-openweb-ui` and it can be changed. ## CSV format Here is a simple example of a CSV file with the knowledge base: ```csv category,subcategory,question,content faq,promotions,loyalty-program,"The company XYZ has a loyalty program when you refer new customers you get a discount on your next purchase, ..." ``` When the `dialog` service starts, it loads the knowledge base into the database through the script called `src/load_csv.py`, so make sure the database is up and paths are correctly configured in the (see [environtment variables](##environment-variables) section). Alternatively, inside `src` folder, run `make load-data path=".csv"`. ### Embedding columns By default, the `load_csv.py` script uses the `content` column to generate the embeddings. To embed more columns together, you can add a environment variable `DIALOG_LOADCSV_EMBED_COLUMNS` in `.env` with desired columns. This is typically the case for examples like Q\&A, where question and answer are in different columns, for example: ``` DIALOG_LOADCSV_EMBED_COLUMNS=question,answer ``` ### Generate an embedding `load_csv.py` Embeddings create a vector representation of a question and answer pair from the knowledge base, enabling semantic search where we look for text passages that are most similar in the vector space. We have a CLI that generates embeddings by reading the knowledge base `csv`. By default, `load_csv.py` performs a **diff** between the existing vector database and the new questions and answers in the `csv`. The **CLI** has some parameters: * `--path`: path to the CSV (knowledge base); * `--cleandb`: deletes all previously imported vectors and reimports everything again. **In Docker** define the environment variable `DIALOG_LOADCSV_CLEARDB` can be set to `true` to enable this option; * `--columns`: defines the columns of the knowledge base (csv) that will be used to generate the embedding. By default, the `content` column is used. ## `.toml` prompt configuration The `[prompt.header]`, `[prompt.suggested]`, and `[fallback.prompt]` fields are mandatory fields used for processing the conversation and connecting to the LLM. The `[prompt.fallback]` field is used when the LLM does not find a compatible embedding in the database; that is, the `[prompt.header]` **is ignored** and the `[prompt.fallback]` is used. Without it, there could be hallucinations about possible answers to questions outside the scope of the embeddings. > In `[prompt.fallback]` the response will be processed by LLM. If you need to return a default message when there is no recommended question in the knowledge base, use the `[prompt.fallback_not_found_relevant_contents]` configuration in the `.toml` *(project configuration)*. It is also possible to add information to the prompt for subcategories and choose some optional LLM parameters like temperature (defaults to 0.2) or model\_name, see below for an example of a complete configuration: ```toml [model] temperature = 0.2 model_name = "gpt-3.5-turbo" [prompt] header = """You are a service operator called Avelino from XYZ, you are an expert in providing qualified service to high-end customers. Be brief in your answers, without being long-winded and objective in your responses. Never say that you are a model (AI), always answer as Avelino. Be polite and friendly!""" suggested = "Here is some possible content that could help the user in a better way." fallback = "I'm sorry, I couldn't find a relevant answer for your question." fallback_not_found_relevant_contents = "I'm sorry, I couldn't find a relevant answer for your question." [prompt.subcategory.loyalty-program] header = """The client is interested in the loyalty program, and needs to be responded to in a salesy way; the loyalty program is our growth strategy.""" ``` > This feature is experimental and may not work as expected, so use it carefully on your environment. ## Monitoring your LLM with LangSmith If you wish to add observability to your LLM application, you may want to use [Langsmith](https://docs.smith.langchain.com/) (so far, for personal use only) to help to debug, test, evaluate, and monitor your chains used in dialog. Follow the [setup instructions](https://docs.smith.langchain.com/setup) and add the env vars into the `.env` file: ``` LANGCHAIN_TRACING_V2=true LANGCHAIN_ENDPOINT="https://api.smith.langchain.com" LANGCHAIN_API_KEY= LANGCHAIN_PROJECT= ``` # Custom LLMs and Plugins In this project, we have a class-based API that allows you to extend the default LLM and add extra methods to your own LLM, enabling better fit for the problem you are trying to solve. ## Extending the default LLM This project is built on top of [dialog-lib](https://github.com/talkdai/dialog-lib), our library that provides a class-based API to create LLM models based on Langchain. If you want to extend the default LLM, you can create a new class that inherits from AbstractLLM (as the default one does, available in the file \`src/dialog/llm/agents/default.py) and override the methods you want to change and add behavior. To use your custom LLM, you just need to add the environment variable LLM\_CLASS to your `.env` file: ``` LLM_CLASS=plugins.my_llm.MyLLMClass ``` ### Is it possible to use LangChain's LCEL as my agent? You can user the default LangChain LCEL approach! It's totally up to you, but the LCEL must return a LangChain `Runnable` object in order to work and this object must be importable from the path you set in the `LLM_CLASS` environment variable. We have an example of an LCEL implementation inside the file `src/dialog/llm/agents/lcel.py`. You can use it by setting the `LLM_CLASS` environment variable to `dialog.llm.agents.lcel.runnable`. ### What happens if my class doesn't implement the AbstractLLM from Dialog Lib or doesn't return a Runnable object? If your class doesn't implement the AbstractLLM or doesn't return a Runnable object, the project will raise an error and load the default agent available in the file `src/dialog/llm/agents/default.py`. ## Adding extra routes to the project Dialog is pretty extensible, being a FastAPI based project allows you to be very creative. ### Adding new models to the project through settings In the release v0.1.3, we enabled users to create multiple endpoints using different models with just a simple tweak in the prompt config config file (the prompt file). The default model is still configured in the same way as the previous versions: you need to define the environment variable 'LLM\_CLASS' to use a model that implements `AbstractLLM` class (any of the models available in dialog-lib implements this class and are ready to use) of your choice and use it on the `/chats/{chat_id}` or `/ask` endpoints. To add a new model, you need to implement a new `[endpoint]` in the toml, just as shown below: ```toml [model] model_name = "gpt-4o" temperature = 0.1 ... some other settings over here ... [[endpoint]] path = "/my-awesome-new-model" model_name = "newmodel" model_class_path = "the.importable.path.to.your.ModelClass" ``` ### Writing a new plugin without a PyPI Package. To add new endpoints or features, you need to create a package inside the `src/plugins` folder and, inside the new package folder, add the following file: * `__init__.py`: the default package initializer from Python (this file can be empty), but we recommend you to create the router here. Inside the `__init__.py` file, you need to create a FastAPI router that will be loaded into the app dynamically: ```python from fastapi import APIRouter router = APIRouter() # add your routes here ``` The variable that instantiates APIRouter must be called **router**. After creating the plugin, to run it, add the environment variable PLUGINS to your .env file: ```bash PLUGINS=plugins.your_plugin_name # or PLUGINS=plugins.your_plugin_name.file_name if there is another file to be used as entrypoint ``` ### WhatsApp Text to Audio Synthesis We already made a WhatsApp plugin that converts the LLM processed output from the message an user sent, into an audio file and sends it back to the user. To use this plugin, you need to clone the [WhatsApp Audio Synth repo](https://github.com/talkdai/whats_audio_synth) inside the plugins folder of this repo and add the following environment variables to your .env file: ```bash WHATSAPP_API_TOKEN= WHATSAPP_ACCOUNT_NUMBER= PLUGINS=plugins.whats_audio_synth.main, ``` ### Available Plugins * [WhatsApp messaging with Audio Synthetizing](https://github.com/talkdai/dialog-whatsapp) * [Sentry](https://github.com/walison17/dialog-sentry/) * [Wati](https://github.com/talkdai/dialog-wati) # talkd/dialog API Development We are very glad that you are interested in developing and helping our project to become successful. ## Good practices We like well organized code (sometimes we fail in this, but we are improving), so in order to help you do a successful PR we are writing this doc. Our coding style try to follow as much as possible from PEP8 and we use `black` and `isort` to fix coding styling issues. ## Local Development setup We've used Python and bundled packages with `poetry`, now it's up to you - ⚠️ we're not yet at the point of explaining in depth how to develop and contribute, [`Makefile`](https://github.com/talkdai/dialog/blob/main/Makefile/README.md) may help you and also reading our [current issues](https://github.com/talkdai/dialog/issues). > **notes:** we recommend using `docker-compose` or our `dev-container` to develop the project or please proceed with setting up the local environment at **your own risk**. ### Creating new/altering tables or columns All of the current available database migrations and tables are handled by `alembic` inside the `dialog-lib` package, so if you need to create a new table or alter a column, you can create a new migration inside this project. ## VS Code Dev Container If you are using VSCode, you can use the [devcontainer](https://github.com/talkdai/dialog/blob/main/.devcontainer/README.md) to run the project. When we upload the environment into devcontainer, we upload the following containers: * `db`: container with the postgres database with **pgvector** extension * `dialog`: container with the api (the project) We don't upload the application when the container is started. To upload the application, run the `make run` command inside the container console (bash). > Remember to generate the embedding vectors and create the `.env` file based on the `.env.sample` file before uploading the application. ```sh make load-data path="know-base-path.csv" make run ``` ## Running Tests Running tests on the project is simple, just use our `test-build` available in Makefile. ```bash make test-build ``` Right now our coverage is pretty limited, we would like to have more code covered as soon as possible. ## Commits Since we are using semantic versioning on the project, we are looking forward for good commit messages and, on those commits, we are expecting to have certain tags on minor or major updates (so we can update the version automatically). ``` - #minor: for minor updates - #major: for major updates - All else will be considered a patch ```