local-llm-agent/lora-adapters/README.md

LoRA Adapters

This directory contains LoRA (Low-Rank Adaptation) adapters for project-specific fine-tuning.

Directory Structure

lora-adapters/
├── README.md           # This file
├── erp-core/           # ERP Core domain adapter
│   ├── adapter_config.json
│   └── adapter_model.safetensors
├── trading/            # Trading platform adapter
│   ├── adapter_config.json
│   └── adapter_model.safetensors
└── {project-name}/     # Additional project adapters
    ├── adapter_config.json
    └── adapter_model.safetensors

Creating LoRA Adapters

Prerequisites

• Base model: mistralai/Mistral-7B-Instruct-v0.2 (or compatible)
• Training data in JSONL format
• PEFT library for training

Training Example

from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load base model
model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2")
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2")

# Configure LoRA
lora_config = LoraConfig(
    r=64,                      # Rank
    lora_alpha=128,            # Alpha scaling
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

# Apply LoRA
model = get_peft_model(model, lora_config)

# Train...
# Save adapter
model.save_pretrained("lora-adapters/your-adapter")

Required Files

1. adapter_config.json: LoRA configuration

   {
     "base_model_name_or_path": "mistralai/Mistral-7B-Instruct-v0.2",
     "peft_type": "LORA",
     "task_type": "CAUSAL_LM",
     "r": 64,
     "lora_alpha": 128,
     "lora_dropout": 0.05,
     "target_modules": ["q_proj", "k_proj", "v_proj", "o_proj"]
   }
   

2. adapter_model.safetensors: LoRA weights

Using LoRA Adapters with vLLM

Configuration

Adapters are automatically mounted in the vLLM container:

# docker-compose.vllm.yml
volumes:
  - ./lora-adapters:/lora-adapters:ro

API Usage

# Chat with LoRA adapter
curl -X POST http://localhost:3160/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistralai/Mistral-7B-Instruct-v0.2",
    "messages": [
      {"role": "user", "content": "How do I create an invoice in the ERP system?"}
    ],
    "lora_adapter": "erp-core"
  }'

Listing Available Adapters

# List LoRA adapters
curl http://localhost:3160/v1/lora/adapters

Project-Specific Adapters

erp-core

• Purpose: ERP domain knowledge (invoices, inventory, accounting)
• Training data: ERP documentation, code, user interactions
• Base model: Mistral-7B-Instruct

trading

• Purpose: Trading platform domain (orders, positions, market data)
• Training data: Trading documentation, API specs, user queries
• Base model: Mistral-7B-Instruct

Best Practices

1. Keep adapters small: LoRA adapters should be < 100MB
2. Test locally first: Verify adapter loads correctly
3. Version control: Track adapter versions separately
4. Documentation: Document training data and hyperparameters

Troubleshooting

Adapter not loading

1. Check file permissions
2. Verify adapter_config.json matches base model
3. Check vLLM logs: docker logs local-llm-vllm

Memory issues

1. Reduce max_loras in docker-compose
2. Use smaller LoRA rank (r=32 instead of r=64)
3. Enable LoRA merging for inference

References

• PEFT Library
• vLLM LoRA Support
• LoRA Paper