From 588aaae5d981fc00f60aa31a510bce191f222dea Mon Sep 17 00:00:00 2001 From: reanon Date: Mon, 21 Apr 2025 08:24:46 +0200 Subject: [PATCH] codebase --- CODEBASE_GUIDE.md | 321 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 321 insertions(+) create mode 100644 CODEBASE_GUIDE.md diff --git a/CODEBASE_GUIDE.md b/CODEBASE_GUIDE.md new file mode 100644 index 0000000..7d88859 --- /dev/null +++ b/CODEBASE_GUIDE.md @@ -0,0 +1,321 @@ +# Project Codebase Guide + +This document serves as a guide and index for the project codebase, designed to help developers and AI agents quickly understand its structure, components, and how to contribute. + +## Table of Contents + +1. [Project Overview](#project-overview) +2. [Directory Structure](#directory-structure) +3. [Core Components](#core-components) + * [Configuration (`src/config.ts`)](#configuration) + * [Server Entry Point (`src/server.ts`)](#server-entry-point) + * [Proxy Layer (`src/proxy/`)](#proxy-layer) + * [User Management (`src/user/`)](#user-management) + * [Admin Interface (`src/admin/`)](#admin-interface) + * [Shared Utilities (`src/shared/`)](#shared-utilities) +4. [Proxy Functionality](#proxy-functionality) + * [Routing (`src/proxy/routes.ts`)](#proxy-routing) + * [Supported Models & Providers](#supported-models--providers) + * [Middleware (`src/proxy/middleware/`)](#proxy-middleware) + * [Adding New Models](#adding-new-models) + * [Adding New APIs/Providers](#adding-new-apisproviders) +5. [Model Management](#model-management) + * [Model Family Definitions](#model-family-definitions) + * [Adding OpenAI Models](#adding-openai-models) + * [Model Mapping & Routing](#model-mapping--routing) + * [Service Information](#service-information) + * [Step-by-Step Guide for Adding a New Model](#step-by-step-guide-for-adding-a-new-model) + * [Model Patterns and Versioning](#model-patterns-and-versioning) + * [Response Format Handling](#response-format-handling) +6. [Key Management](#key-management) + * [Key Pool System](#key-pool-system) + * [Provider-Specific Key Management](#provider-specific-key-management) + * [Key Rotation and Health Checks](#key-rotation-and-health-checks) +7. [Data Management](#data-management) + * [Database (`src/shared/database/`)](#database) + * [File Storage (`src/shared/file-storage/`)](#file-storage) +8. [Authentication & Authorization](#authentication--authorization) +9. [Logging & Monitoring](#logging--monitoring) +10. [Deployment](#deployment) +11. [Contributing](#contributing) + +## Project Overview + +This project provides a proxy layer for various Large Language Models (LLMs) and potentially other AI APIs. It aims to offer a unified interface, manage API keys securely, handle rate limiting, usage tracking, and potentially add features like response caching or prompt modification. + +## Directory Structure + +``` +. +├── .env.example # Example environment variables +├── .gitattributes # Git attributes +├── .gitignore # Git ignore rules +├── .husky/ # Git hooks +├── .prettierrc # Code formatting rules +├── CODEBASE_GUIDE.md # This file +├── README.md # Project README +├── data/ # Data files (e.g., SQLite DB) +├── docker/ # Docker configuration +├── docs/ # Documentation files +├── http-client.env.json # HTTP client environment +├── package-lock.json # NPM lock file +├── package.json # Project dependencies and scripts +├── patches/ # Patches for dependencies +├── public/ # Static assets served by the web server +├── render.yaml # Render deployment configuration +├── scripts/ # Utility scripts +├── src/ # Source code +│ ├── admin/ # Admin interface logic +│ ├── config.ts # Application configuration +│ ├── info-page.ts # Logic for the info page +│ ├── logger.ts # Logging setup +│ ├── proxy/ # Core proxy logic for different providers +│ ├── server.ts # Express server setup and main entry point +│ ├── service-info.ts # Service information logic +│ ├── shared/ # Shared utilities, types, and modules +│ └── user/ # User management logic +├── tsconfig.json # TypeScript configuration +``` + +## Core Components + +### Configuration (`src/config.ts`) + +* Loads environment variables and defines application settings. +* Contains configuration for database connections, API keys (placeholders/retrieval methods), logging levels, rate limits, etc. +* Uses `dotenv` and potentially a schema validation library (like Zod) to ensure required variables are present. + +### Server Entry Point (`src/server.ts`) + +* Initializes the Express application. +* Sets up core middleware (e.g., body parsing, CORS, logging). +* Mounts routers for different parts of the application (admin, user, proxy). +* Starts the HTTP server. + +### Proxy Layer (`src/proxy/`) + +* The heart of the application, handling requests to downstream AI APIs. +* Contains individual modules for each supported provider (e.g., `openai.ts`, `anthropic.ts`). +* Handles request transformation, authentication against the target API, and response handling. +* Uses middleware for common proxy tasks. + +### User Management (`src/user/`) + +* Handles user registration, login, session management, and potentially API key generation/management for end-users. +* Likely interacts with the database (`src/shared/database/`). + +### Admin Interface (`src/admin/`) + +* Provides an interface for administrators to manage users, monitor usage, configure settings, etc. +* May have its own set of routes and views. + +### Shared Utilities (`src/shared/`) + +* Contains reusable code across different modules. + * `api-schemas/`: Zod schemas for API request/response validation. + * `database/`: Database connection, schemas (e.g., Prisma), and query logic. + * `errors.ts`: Custom error classes. + * `key-management/`: Logic for managing API keys (if applicable). + * `models.ts`: Core data models/types used throughout the application. + * `prompt-logging/`: Logic for logging prompts and responses. + * `tokenization/`: Utilities for counting tokens. + * `utils.ts`: General utility functions. + +## Proxy Functionality + +### Proxy Routing (`src/proxy/routes.ts`) + +* Defines the API endpoints for the proxy service (e.g., `/v1/chat/completions`). +* Maps incoming requests to the appropriate provider-specific handler based on the request path, headers, or body content (e.g., model requested). +* Applies relevant middleware (authentication, rate limiting, queuing, etc.). + +### Supported Models & Providers + +* **OpenAI:** Handled in `src/proxy/openai.ts`. Supports models like GPT-4, GPT-3.5-turbo, as well as o-series models (o1, o1-mini, o1-pro, o3, o3-mini, o4-mini). Handles chat completions and potentially image generation (`src/proxy/openai-image.ts`). +* **Anthropic:** Handled in `src/proxy/anthropic.ts`. Supports Claude models. May use AWS Bedrock (`src/proxy/aws-claude.ts`) or Anthropic's direct API. +* **Google AI / Vertex AI:** Handled in `src/proxy/google-ai.ts` and `src/proxy/gcp.ts`. Supports Gemini models (gemini-flash, gemini-pro, gemini-ultra). +* **Mistral AI:** Handled in `src/proxy/mistral-ai.ts`. Supports Mistral models via their API or potentially AWS (`src/proxy/aws-mistral.ts`). +* **Azure OpenAI:** Handled in `src/proxy/azure.ts`. Provides an alternative endpoint for OpenAI models via Azure. +* **Deepseek:** Handled in `src/proxy/deepseek.ts`. +* **Xai:** Handled in `src/proxy/xai.ts`. +* **AWS (General):** `src/proxy/aws.ts` might contain shared AWS logic (e.g., authentication). + +### Middleware (`src/proxy/middleware/`) + +* **`gatekeeper.ts`:** Likely handles initial request validation, authentication, and authorization checks before hitting provider logic. Checks origin (`check-origin.ts`), potentially custom tokens (`check-risu-token.ts`). +* **`rate-limit.ts`:** Implements rate limiting logic, potentially per-user or per-key. +* **`queue.ts`:** Manages request queuing, possibly to handle concurrency limits or prioritize requests. + +### Adding New Models + +1. **Identify the Provider:** Determine if the new model belongs to an existing provider (e.g., a new OpenAI model) or a new one. +2. **Update Provider Logic (if existing):** + * Modify the relevant provider file (e.g., `src/proxy/openai.ts`). + * Update model lists or logic that selects/validates models. + * Adjust any request/response transformations if the new model has a different API schema. + * Update model information in shared files like `src/shared/models.ts` if necessary. +3. **Update Routing (if necessary):** Modify `src/proxy/routes.ts` if the new model requires a different endpoint or routing logic. +4. **Configuration:** Add any new API keys or configuration parameters to `.env.example` and `src/config.ts`. +5. **Testing:** Add unit or integration tests for the new model. + +### Adding New APIs/Providers + +1. **Create Provider Module:** Create a new file in `src/proxy/` (e.g., `src/proxy/new-provider.ts`). +2. **Implement Handler:** + * Write the core logic to handle requests for this provider. This typically involves: + * Receiving the standardized request from the router. + * Transforming the request into the format expected by the new provider's API. + * Authenticating with the new provider's API (fetching keys from config). + * Making the API call (consider using a robust HTTP client like `axios` or `node-fetch`). + * Handling streaming responses if applicable (using helpers from `src/shared/streaming.ts`). + * Transforming the provider's response back into a standardized format. + * Handling errors gracefully. +3. **Add Routing:** + * Import the new handler in `src/proxy/routes.ts`. + * Add new routes or modify existing routing logic to direct requests to the new handler based on model name, path, or other criteria. + * Apply necessary middleware (gatekeeper, rate limiter, queue). +4. **Create Key Management:** + * Create a new directory in `src/shared/key-management/` for the provider. + * Implement provider-specific key management (key checkers, token counters). +5. **Configuration:** + * Add configuration variables (API keys, base URLs) to `.env.example` and `src/config.ts`. + * Update `src/config.ts` to load and validate the new variables. +6. **Model Information:** Add details about the new provider and its models to `src/shared/models.ts` or similar shared locations. +7. **Tokenization (if applicable):** If token counting is needed, add or update tokenization logic in `src/shared/tokenization/`. +8. **Testing:** Implement thorough tests for the new provider integration. +9. **Documentation:** Update this guide and any other relevant documentation. + +## Model Management + +### Model Family Definitions + +* **Model Family Definitions:** The project uses a family-based approach to group similar models together. These are defined in `src/shared/models.ts`. +* Each model is part of a model family (e.g., "gpt4", "claude", "gemini-pro") which helps with routing, key management, and feature support. +* The `MODEL_FAMILIES` array contains all supported model families, and the `MODEL_FAMILY_SERVICE` mapping connects each family to its provider service. + +### Adding OpenAI Models + +When adding new OpenAI models to the codebase, there are several files that must be updated: + +1. **Update Model Types (`src/shared/models.ts`):** + - Add the new model to the `OpenAIModelFamily` type + - Add the model to the `MODEL_FAMILIES` array + - Add the Azure variants for the model if applicable + - Add the model to `MODEL_FAMILY_SERVICE` mapping + - Update `OPENAI_MODEL_FAMILY_MAP` with regex patterns to match the model names + +2. **Update Context Size Limits (`src/proxy/middleware/request/preprocessors/validate-context-size.ts`):** + - Add regex matching for the new model + - Set the appropriate context token limit for the model + +3. **Update Token Cost Tracking (`src/shared/stats.ts`):** + - Add pricing information for the new model in the `getTokenCostUsd` function + - Include both input and output prices in the comments for clarity + +4. **Update Feature Support Checks (`src/proxy/openai.ts`):** + - If the model supports special features like the reasoning API parameter (`isO1Model` function), update the appropriate function + - For model feature detection, prefer using regex patterns over explicit lists when possible, as this handles date-stamped versions better + +5. **Update Display Names (`src/info-page.ts`):** + - Add friendly display names for the new models in the `MODEL_FAMILY_FRIENDLY_NAME` object + +6. **Update Key Management Provider Files:** + - For OpenAI keys in `src/shared/key-management/openai/provider.ts`, add token counters for the new models + - For Azure OpenAI keys in `src/shared/key-management/azure/provider.ts`, add token counters for the Azure versions + +### Model Patterns and Versioning + +The codebase handles several patterns for model naming and versioning: + +1. **Date-stamped Models:** Many models include date stamps (e.g., `gpt-4-0125-preview`). The regex patterns in `OPENAI_MODEL_FAMILY_MAP` account for these with patterns like `^gpt-4o(-\\d{4}-\\d{2}-\\d{2})?$`. + +2. **O-Series Models:** OpenAI's o-series models (o1, o1-mini, o1-pro, o3, o3-mini, o4-mini) follow a different naming convention. The codebase handles these with dedicated model families and regex patterns. + +3. **Preview/Non-Preview Variants:** Some models have preview variants (e.g., `gpt-4.5-preview`). The regex patterns in `OPENAI_MODEL_FAMILY_MAP` account for these with patterns like `^gpt-4\\.5(-preview)?(-\\d{4}-\\d{2}-\\d{2})?$`. + +When adding new models, try to follow the existing patterns for consistency. + +### Response Format Handling + +The codebase includes special handling for different API response formats: + +1. **Chat vs. Text Completions:** There's transformation logic in `openai.ts` to convert between chat completions and text completions formats (`transformTurboInstructResponse`). + +2. **Newer API Formats:** For newer APIs like the Responses API, there's transformation logic (`transformResponsesApiResponse`) to convert responses to a format compatible with existing clients. + +When adding support for new models or APIs, consider whether transformation is needed to maintain compatibility with existing clients. + +## Key Management + +### Key Pool System + +The project uses a sophisticated key pool system (`src/shared/key-management/key-pool.ts`) to manage API keys for different providers. Key features include: + +* **Key Selection:** The system selects the appropriate key based on model family, region preferences, and other criteria. +* **Rotation:** Keys are rotated to distribute usage and avoid hitting rate limits. +* **Health Checks:** Keys are checked periodically to ensure they're still valid and within rate limits. + +### Provider-Specific Key Management + +Each provider has its own key management module in `src/shared/key-management/`: + +* **Key Checkers:** Each provider implements key checkers to validate keys and check their status. +* **Token Counters:** Providers implement token counting logic specific to their pricing model. +* **Models Support:** Keys are associated with specific model families they support. + +When adding a new model or provider, you'll need to update or create the appropriate key management files. + +### Key Rotation and Health Checks + +The key pool system includes logic for: + +* **Rotation Strategy:** Keys are selected based on a prioritization strategy (`prioritize-keys.ts`). +* **Disabling Unhealthy Keys:** Keys that fail health checks are temporarily disabled. +* **Rate Limit Awareness:** The system tracks usage to avoid hitting provider rate limits. + +## Data Management + +### Database (`src/shared/database/`) + +* Likely uses Prisma or a similar ORM. +* Defines database schemas (e.g., for users, API keys, usage logs). +* Provides functions for interacting with the database. +* Configuration is managed in `src/config.ts`. + +### File Storage (`src/shared/file-storage/`) + +* May be used for storing logs, cached data, or user-uploaded files. +* Could integrate with local storage or cloud providers (e.g., S3, GCS). + +## Authentication & Authorization + +* **User Auth:** Handled in `src/user/` potentially using sessions (`src/shared/with-session.ts`) or JWTs. +* **Proxy Auth:** The `gatekeeper.ts` middleware likely verifies incoming requests to the proxy endpoints. This could involve checking: + * Custom API keys stored in the database (`src/shared/database/`). + * Specific tokens (`check-risu-token.ts`). + * HMAC signatures (`src/shared/hmac-signing.ts`). + * Origin checks (`check-origin.ts`). +* **Downstream Auth:** Each provider module (`src/proxy/*.ts`) handles authentication with the actual AI service API using keys from the configuration. + +## Logging & Monitoring + +* **Logging:** Configured in `src/logger.ts`, likely using a library like `pino` or `winston`. Logs requests, errors, and important events. +* **Prompt Logging:** Specific logic for logging prompts and responses might exist in `src/shared/prompt-logging/`. +* **Stats/Monitoring:** `src/shared/stats.ts` might handle collecting and exposing application metrics. + +## Deployment + +* **Docker:** The project likely includes Docker configuration for containerized deployment. +* **Render:** The `render.yaml` file suggests the project is or can be deployed on Render. +* **Environment Variables:** The `.env.example` file provides a template for required environment variables in production. + +## Contributing + +When contributing to this project: + +1. **Follow Coding Standards:** Use the established patterns and standards in the codebase. The `.prettierrc` file defines code formatting rules. +2. **Update Documentation:** Keep this guide updated when adding new components or changing existing ones. +3. **Add Tests:** Ensure your changes are tested appropriately. +4. **Update Configuration:** If your changes require new environment variables, update `.env.example`. + +*This guide provides a high-level overview. For detailed information, refer to the specific source code files.*