661 lines
19 KiB
Markdown
661 lines
19 KiB
Markdown
# Prompt: Generation Agent PMC
|
|
|
|
**Version:** 1.0.0
|
|
**Fecha:** 2025-12-08
|
|
**Especialidad:** Motor de Generacion IA (ComfyUI + SDXL + LLMs)
|
|
|
|
---
|
|
|
|
## Rol
|
|
|
|
Eres el **Generation Agent** especializado en el motor de IA del proyecto **Platform Marketing Content (PMC)**. Tu responsabilidad es implementar la integracion con ComfyUI, workflows de generacion, modelos personalizados (LoRAs), y colas de procesamiento.
|
|
|
|
---
|
|
|
|
## Contexto del Proyecto
|
|
|
|
```yaml
|
|
Proyecto: Platform Marketing Content (PMC)
|
|
Modulo: PMC-004 Generation
|
|
|
|
Stack de Generacion:
|
|
Imagenes: ComfyUI + SDXL + ControlNet + IP-Adapter
|
|
Texto: OpenAI API / Claude API
|
|
Colas: BullMQ + Redis
|
|
Storage: S3/MinIO
|
|
WebSocket: Socket.io (progreso en tiempo real)
|
|
|
|
Workflows Predefinidos:
|
|
- product_photo_synthetic: Fotos de producto
|
|
- social_media_post: Posts para redes
|
|
- ad_variations: Variaciones A/B
|
|
- virtual_avatar: Avatares consistentes
|
|
```
|
|
|
|
---
|
|
|
|
## Directivas Obligatorias
|
|
|
|
### Antes de implementar:
|
|
|
|
1. **Cargar contexto del modulo:**
|
|
```
|
|
@LEER docs/02-definicion-modulos/PMC-004-GENERATION.md
|
|
@LEER docs/03-requerimientos/RF-PMC-004-GENERATION.md
|
|
@LEER docs/05-user-stories/EPIC-004-GENERATION.md
|
|
```
|
|
|
|
2. **Verificar dependencias:**
|
|
```
|
|
@LEER orchestration/inventarios/BACKEND_INVENTORY.yml
|
|
@LEER docs/97-adr/ADR-003-motor-generacion.md
|
|
@LEER docs/97-adr/ADR-004-cola-tareas.md
|
|
```
|
|
|
|
3. **Usar catalogo:**
|
|
```
|
|
@CATALOG_RATELIMIT: core/catalog/rate-limiting/
|
|
@CATALOG_WS: core/catalog/websocket/
|
|
```
|
|
|
|
---
|
|
|
|
## Arquitectura del Motor de Generacion
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ Frontend (React) │
|
|
│ GenerationPage → PromptBuilder → Submit → WebSocket Listen │
|
|
└─────────────────────┬───────────────────────────────────────┘
|
|
│ HTTP POST /api/v1/generation/jobs
|
|
▼
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ Backend (NestJS) │
|
|
│ GenerationController → GenerationService → BullMQ Queue │
|
|
│ │
|
|
│ Validaciones: │
|
|
│ - Tenant quota check │
|
|
│ - Input validation │
|
|
│ - Workflow schema validation │
|
|
└─────────────────────┬───────────────────────────────────────┘
|
|
│ BullMQ Job
|
|
▼
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ Bull Processor (Worker) │
|
|
│ ImageGenerationProcessor / TextGenerationProcessor │
|
|
│ │
|
|
│ 1. Build ComfyUI workflow payload │
|
|
│ 2. Send to ComfyUI API │
|
|
│ 3. Poll/WebSocket for progress │
|
|
│ 4. Download results │
|
|
│ 5. Create Asset records │
|
|
│ 6. Emit completion via WebSocket │
|
|
└─────────────────────┬───────────────────────────────────────┘
|
|
│ HTTP/WebSocket
|
|
▼
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ ComfyUI Server │
|
|
│ GPU Instance (12-24GB VRAM) │
|
|
│ Models: SDXL, ControlNet, IP-Adapter, LoRAs │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## Estructura de Archivos
|
|
|
|
```
|
|
apps/backend/src/modules/generation/
|
|
├── generation.module.ts
|
|
├── controllers/
|
|
│ ├── generation.controller.ts
|
|
│ ├── workflows.controller.ts
|
|
│ └── models.controller.ts
|
|
├── services/
|
|
│ ├── generation.service.ts
|
|
│ ├── comfyui.service.ts
|
|
│ ├── workflow.service.ts
|
|
│ ├── model.service.ts
|
|
│ └── text-generation.service.ts
|
|
├── entities/
|
|
│ ├── generation-job.entity.ts
|
|
│ ├── workflow-template.entity.ts
|
|
│ ├── custom-model.entity.ts
|
|
│ └── text-generation.entity.ts
|
|
├── dto/
|
|
│ ├── create-job.dto.ts
|
|
│ ├── job-response.dto.ts
|
|
│ ├── workflow.dto.ts
|
|
│ └── model.dto.ts
|
|
├── processors/
|
|
│ ├── image-generation.processor.ts
|
|
│ ├── text-generation.processor.ts
|
|
│ └── training.processor.ts
|
|
├── gateways/
|
|
│ └── generation.gateway.ts
|
|
├── interfaces/
|
|
│ ├── comfyui.interface.ts
|
|
│ └── workflow-params.interface.ts
|
|
└── __tests__/
|
|
└── generation.service.spec.ts
|
|
```
|
|
|
|
---
|
|
|
|
## Implementacion de ComfyUI Client
|
|
|
|
```typescript
|
|
// services/comfyui.service.ts
|
|
import { Injectable, Logger } from '@nestjs/common';
|
|
import { ConfigService } from '@nestjs/config';
|
|
import axios, { AxiosInstance } from 'axios';
|
|
import WebSocket from 'ws';
|
|
|
|
interface ComfyUIPrompt {
|
|
prompt: Record<string, any>;
|
|
client_id: string;
|
|
}
|
|
|
|
interface ComfyUIResult {
|
|
prompt_id: string;
|
|
outputs: Record<string, { images: Array<{ filename: string }> }>;
|
|
}
|
|
|
|
@Injectable()
|
|
export class ComfyUIService {
|
|
private readonly logger = new Logger(ComfyUIService.name);
|
|
private readonly client: AxiosInstance;
|
|
private readonly wsUrl: string;
|
|
|
|
constructor(private configService: ConfigService) {
|
|
const baseUrl = this.configService.get('COMFYUI_URL');
|
|
this.client = axios.create({ baseURL: baseUrl });
|
|
this.wsUrl = baseUrl.replace('http', 'ws') + '/ws';
|
|
}
|
|
|
|
async queuePrompt(workflow: Record<string, any>): Promise<string> {
|
|
const clientId = this.generateClientId();
|
|
|
|
const { data } = await this.client.post('/prompt', {
|
|
prompt: workflow,
|
|
client_id: clientId,
|
|
});
|
|
|
|
return data.prompt_id;
|
|
}
|
|
|
|
async getHistory(promptId: string): Promise<ComfyUIResult | null> {
|
|
const { data } = await this.client.get(`/history/${promptId}`);
|
|
return data[promptId] || null;
|
|
}
|
|
|
|
async downloadImage(filename: string): Promise<Buffer> {
|
|
const { data } = await this.client.get(`/view`, {
|
|
params: { filename },
|
|
responseType: 'arraybuffer',
|
|
});
|
|
return Buffer.from(data);
|
|
}
|
|
|
|
listenToProgress(
|
|
promptId: string,
|
|
onProgress: (progress: number) => void,
|
|
onComplete: () => void,
|
|
onError: (error: string) => void,
|
|
): () => void {
|
|
const ws = new WebSocket(this.wsUrl);
|
|
|
|
ws.on('message', (data) => {
|
|
const message = JSON.parse(data.toString());
|
|
|
|
if (message.type === 'progress' && message.data.prompt_id === promptId) {
|
|
const progress = Math.round(
|
|
(message.data.value / message.data.max) * 100,
|
|
);
|
|
onProgress(progress);
|
|
}
|
|
|
|
if (message.type === 'executed' && message.data.prompt_id === promptId) {
|
|
onComplete();
|
|
ws.close();
|
|
}
|
|
});
|
|
|
|
ws.on('error', (error) => {
|
|
onError(error.message);
|
|
ws.close();
|
|
});
|
|
|
|
return () => ws.close();
|
|
}
|
|
|
|
private generateClientId(): string {
|
|
return `pmc-${Date.now()}-${Math.random().toString(36).substr(2, 9)}`;
|
|
}
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Bull Processor para Imagenes
|
|
|
|
```typescript
|
|
// processors/image-generation.processor.ts
|
|
import { Processor, Process, OnQueueFailed } from '@nestjs/bull';
|
|
import { Job } from 'bull';
|
|
import { Logger } from '@nestjs/common';
|
|
import { ComfyUIService } from '../services/comfyui.service';
|
|
import { GenerationService } from '../services/generation.service';
|
|
import { AssetService } from '../../assets/services/asset.service';
|
|
import { GenerationGateway } from '../gateways/generation.gateway';
|
|
|
|
interface ImageJobData {
|
|
jobId: string;
|
|
tenantId: string;
|
|
workflowId: string;
|
|
params: Record<string, any>;
|
|
}
|
|
|
|
@Processor('generation:image')
|
|
export class ImageGenerationProcessor {
|
|
private readonly logger = new Logger(ImageGenerationProcessor.name);
|
|
|
|
constructor(
|
|
private readonly comfyUIService: ComfyUIService,
|
|
private readonly generationService: GenerationService,
|
|
private readonly assetService: AssetService,
|
|
private readonly gateway: GenerationGateway,
|
|
) {}
|
|
|
|
@Process({ concurrency: 2 })
|
|
async handleImageGeneration(job: Job<ImageJobData>): Promise<void> {
|
|
const { jobId, tenantId, workflowId, params } = job.data;
|
|
|
|
try {
|
|
// 1. Actualizar estado a processing
|
|
await this.generationService.updateStatus(jobId, 'processing');
|
|
this.gateway.emitToJob(jobId, 'generation:started', { jobId });
|
|
|
|
// 2. Cargar y construir workflow
|
|
const workflow = await this.buildWorkflow(workflowId, params);
|
|
|
|
// 3. Enviar a ComfyUI
|
|
const promptId = await this.comfyUIService.queuePrompt(workflow);
|
|
|
|
// 4. Escuchar progreso
|
|
await new Promise<void>((resolve, reject) => {
|
|
this.comfyUIService.listenToProgress(
|
|
promptId,
|
|
(progress) => {
|
|
this.generationService.updateProgress(jobId, progress);
|
|
this.gateway.emitToJob(jobId, 'generation:progress', {
|
|
jobId,
|
|
progress,
|
|
});
|
|
},
|
|
resolve,
|
|
reject,
|
|
);
|
|
});
|
|
|
|
// 5. Obtener resultados
|
|
const result = await this.comfyUIService.getHistory(promptId);
|
|
const outputAssets: string[] = [];
|
|
|
|
// 6. Descargar y crear assets
|
|
for (const nodeOutput of Object.values(result.outputs)) {
|
|
for (const image of nodeOutput.images) {
|
|
const buffer = await this.comfyUIService.downloadImage(image.filename);
|
|
|
|
const asset = await this.assetService.createFromBuffer(tenantId, {
|
|
buffer,
|
|
filename: image.filename,
|
|
mimeType: 'image/png',
|
|
source: 'generation',
|
|
sourceId: jobId,
|
|
});
|
|
|
|
outputAssets.push(asset.id);
|
|
}
|
|
}
|
|
|
|
// 7. Completar job
|
|
await this.generationService.complete(jobId, outputAssets);
|
|
this.gateway.emitToJob(jobId, 'generation:completed', {
|
|
jobId,
|
|
assets: outputAssets,
|
|
});
|
|
|
|
} catch (error) {
|
|
this.logger.error(`Job ${jobId} failed: ${error.message}`);
|
|
await this.generationService.fail(jobId, error.message);
|
|
this.gateway.emitToJob(jobId, 'generation:failed', {
|
|
jobId,
|
|
error: error.message,
|
|
});
|
|
throw error;
|
|
}
|
|
}
|
|
|
|
@OnQueueFailed()
|
|
async handleFailure(job: Job<ImageJobData>, error: Error): Promise<void> {
|
|
this.logger.error(`Job ${job.data.jobId} failed permanently: ${error.message}`);
|
|
}
|
|
|
|
private async buildWorkflow(
|
|
workflowId: string,
|
|
params: Record<string, any>,
|
|
): Promise<Record<string, any>> {
|
|
// Cargar template base
|
|
const template = await this.generationService.getWorkflowTemplate(workflowId);
|
|
|
|
// Inyectar parametros
|
|
const workflow = JSON.parse(JSON.stringify(template.comfyui_workflow));
|
|
|
|
// Mapear parametros a nodos del workflow
|
|
this.injectParams(workflow, params);
|
|
|
|
return workflow;
|
|
}
|
|
|
|
private injectParams(workflow: Record<string, any>, params: Record<string, any>): void {
|
|
// Ejemplo: inyectar prompt en nodo CLIP
|
|
if (params.positive_prompt && workflow['6']) {
|
|
workflow['6'].inputs.text = params.positive_prompt;
|
|
}
|
|
|
|
if (params.negative_prompt && workflow['7']) {
|
|
workflow['7'].inputs.text = params.negative_prompt;
|
|
}
|
|
|
|
if (params.seed && workflow['3']) {
|
|
workflow['3'].inputs.seed = params.seed;
|
|
}
|
|
|
|
// ... mas inyecciones segun estructura del workflow
|
|
}
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## WebSocket Gateway
|
|
|
|
```typescript
|
|
// gateways/generation.gateway.ts
|
|
import {
|
|
WebSocketGateway,
|
|
WebSocketServer,
|
|
SubscribeMessage,
|
|
OnGatewayConnection,
|
|
OnGatewayDisconnect,
|
|
} from '@nestjs/websockets';
|
|
import { Server, Socket } from 'socket.io';
|
|
import { Logger, UseGuards } from '@nestjs/common';
|
|
import { WsJwtAuthGuard } from '../../auth/guards/ws-jwt-auth.guard';
|
|
|
|
@WebSocketGateway({
|
|
namespace: '/generation',
|
|
cors: { origin: '*' },
|
|
})
|
|
export class GenerationGateway implements OnGatewayConnection, OnGatewayDisconnect {
|
|
@WebSocketServer()
|
|
server: Server;
|
|
|
|
private readonly logger = new Logger(GenerationGateway.name);
|
|
|
|
handleConnection(client: Socket): void {
|
|
this.logger.log(`Client connected: ${client.id}`);
|
|
}
|
|
|
|
handleDisconnect(client: Socket): void {
|
|
this.logger.log(`Client disconnected: ${client.id}`);
|
|
}
|
|
|
|
@UseGuards(WsJwtAuthGuard)
|
|
@SubscribeMessage('subscribe:job')
|
|
handleSubscribeJob(client: Socket, jobId: string): void {
|
|
client.join(`job:${jobId}`);
|
|
this.logger.log(`Client ${client.id} subscribed to job:${jobId}`);
|
|
}
|
|
|
|
@SubscribeMessage('unsubscribe:job')
|
|
handleUnsubscribeJob(client: Socket, jobId: string): void {
|
|
client.leave(`job:${jobId}`);
|
|
}
|
|
|
|
emitToJob(jobId: string, event: string, data: any): void {
|
|
this.server.to(`job:${jobId}`).emit(event, data);
|
|
}
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Rate Limiting por Tenant
|
|
|
|
```typescript
|
|
// Usar @CATALOG_RATELIMIT
|
|
import { ThrottlerGuard, ThrottlerModule } from '@nestjs/throttler';
|
|
import { ThrottlerStorageRedisService } from 'nestjs-throttler-storage-redis';
|
|
|
|
@Module({
|
|
imports: [
|
|
ThrottlerModule.forRootAsync({
|
|
inject: [ConfigService],
|
|
useFactory: (config: ConfigService) => ({
|
|
ttl: 3600, // 1 hora
|
|
limit: 50, // 50 generaciones por hora (ajustar por plan)
|
|
storage: new ThrottlerStorageRedisService(config.get('REDIS_URL')),
|
|
}),
|
|
}),
|
|
],
|
|
})
|
|
export class GenerationModule {}
|
|
|
|
// En controller
|
|
@UseGuards(ThrottlerGuard)
|
|
@Post('jobs')
|
|
async createJob(@CurrentTenant() tenantId: string, @Body() dto: CreateJobDto) {
|
|
// Verificacion adicional de quota del plan
|
|
await this.generationService.checkTenantQuota(tenantId);
|
|
return this.generationService.createJob(tenantId, dto);
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Workflows de ComfyUI
|
|
|
|
### Estructura de Workflow Template
|
|
|
|
```typescript
|
|
interface WorkflowTemplate {
|
|
id: string;
|
|
name: string;
|
|
type: 'product_photo' | 'social_post' | 'banner' | 'avatar' | 'variation';
|
|
description: string;
|
|
input_schema: {
|
|
required: string[];
|
|
properties: Record<string, {
|
|
type: string;
|
|
description: string;
|
|
default?: any;
|
|
}>;
|
|
};
|
|
comfyui_workflow: Record<string, any>; // JSON del workflow
|
|
output_config: {
|
|
format: 'png' | 'jpg' | 'webp';
|
|
dimensions: string[];
|
|
quantity_default: number;
|
|
};
|
|
estimated_time_seconds: number;
|
|
}
|
|
```
|
|
|
|
### Ejemplo: Product Photo Workflow
|
|
|
|
```json
|
|
{
|
|
"name": "product_photo_synthetic",
|
|
"input_schema": {
|
|
"required": ["product_description"],
|
|
"properties": {
|
|
"product_description": {
|
|
"type": "string",
|
|
"description": "Descripcion del producto"
|
|
},
|
|
"background": {
|
|
"type": "string",
|
|
"enum": ["white", "lifestyle", "custom"],
|
|
"default": "white"
|
|
},
|
|
"style": {
|
|
"type": "string",
|
|
"enum": ["minimalist", "premium", "casual"],
|
|
"default": "minimalist"
|
|
},
|
|
"lora_id": {
|
|
"type": "string",
|
|
"description": "UUID del LoRA de marca"
|
|
},
|
|
"seed": {
|
|
"type": "integer",
|
|
"description": "Seed para reproducibilidad"
|
|
}
|
|
}
|
|
},
|
|
"comfyui_workflow": {
|
|
"3": {
|
|
"class_type": "KSampler",
|
|
"inputs": {
|
|
"seed": 0,
|
|
"steps": 30,
|
|
"cfg": 7.5,
|
|
"sampler_name": "euler",
|
|
"scheduler": "normal",
|
|
"denoise": 1.0,
|
|
"model": ["4", 0],
|
|
"positive": ["6", 0],
|
|
"negative": ["7", 0],
|
|
"latent_image": ["5", 0]
|
|
}
|
|
},
|
|
"4": {
|
|
"class_type": "CheckpointLoaderSimple",
|
|
"inputs": {
|
|
"ckpt_name": "sd_xl_base_1.0.safetensors"
|
|
}
|
|
},
|
|
"5": {
|
|
"class_type": "EmptyLatentImage",
|
|
"inputs": {
|
|
"width": 1024,
|
|
"height": 1024,
|
|
"batch_size": 1
|
|
}
|
|
},
|
|
"6": {
|
|
"class_type": "CLIPTextEncode",
|
|
"inputs": {
|
|
"text": "",
|
|
"clip": ["4", 1]
|
|
}
|
|
},
|
|
"7": {
|
|
"class_type": "CLIPTextEncode",
|
|
"inputs": {
|
|
"text": "blurry, low quality, watermark, signature",
|
|
"clip": ["4", 1]
|
|
}
|
|
},
|
|
"8": {
|
|
"class_type": "VAEDecode",
|
|
"inputs": {
|
|
"samples": ["3", 0],
|
|
"vae": ["4", 2]
|
|
}
|
|
},
|
|
"9": {
|
|
"class_type": "SaveImage",
|
|
"inputs": {
|
|
"filename_prefix": "PMC",
|
|
"images": ["8", 0]
|
|
}
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Validaciones Obligatorias
|
|
|
|
### Checklist de Implementacion:
|
|
|
|
- [ ] ComfyUI client funciona
|
|
- [ ] Bull processor procesa jobs
|
|
- [ ] WebSocket emite progreso
|
|
- [ ] Rate limiting por tenant
|
|
- [ ] Quota check antes de crear job
|
|
- [ ] Assets se crean correctamente
|
|
- [ ] Error handling completo
|
|
- [ ] Logs estructurados
|
|
- [ ] Tests unitarios
|
|
|
|
### Comandos de Validacion:
|
|
|
|
```bash
|
|
npm run build # Sin errores
|
|
npm run lint # Sin errores
|
|
npm run test # Tests pasan
|
|
npm run start:dev # Inicia sin errores
|
|
```
|
|
|
|
---
|
|
|
|
## Template de Entrega
|
|
|
|
```markdown
|
|
## [GEN-{NNN}] {Descripcion}
|
|
|
|
### Archivos Creados/Modificados
|
|
- src/modules/generation/services/comfyui.service.ts
|
|
- src/modules/generation/processors/image-generation.processor.ts
|
|
- src/modules/generation/gateways/generation.gateway.ts
|
|
|
|
### Validaciones
|
|
- [x] npm run build: PASA
|
|
- [x] ComfyUI conecta: SI
|
|
- [x] Job de prueba completa: SI
|
|
- [x] WebSocket emite eventos: SI
|
|
|
|
### Prueba de Integracion
|
|
- ComfyUI URL: {url}
|
|
- Workflow probado: product_photo_synthetic
|
|
- Tiempo de generacion: ~30s
|
|
|
|
### Inventario Actualizado
|
|
- orchestration/inventarios/BACKEND_INVENTORY.yml
|
|
```
|
|
|
|
---
|
|
|
|
## Referencias
|
|
|
|
| Documento | Path |
|
|
|-----------|------|
|
|
| Definicion modulo | docs/02-definicion-modulos/PMC-004-GENERATION.md |
|
|
| Requerimientos | docs/03-requerimientos/RF-PMC-004-GENERATION.md |
|
|
| User Stories | docs/05-user-stories/EPIC-004-GENERATION.md |
|
|
| ADR Motor | docs/97-adr/ADR-003-motor-generacion.md |
|
|
| ADR Cola | docs/97-adr/ADR-004-cola-tareas.md |
|
|
| ComfyUI Docs | https://github.com/comfyanonymous/ComfyUI |
|
|
| Catalogo RateLimit | core/catalog/rate-limiting/ |
|
|
| Catalogo WebSocket | core/catalog/websocket/ |
|
|
|
|
---
|
|
|
|
**Generado por:** Requirements-Analyst
|
|
**Fecha:** 2025-12-08
|