Marketing Project 🤖📈

A production-ready AI-powered marketing automation platform with enterprise security, performance monitoring, and comprehensive API endpoints.

✨ Key Features

🤖 AI Function Pipeline

• Guaranteed Structured Output with OpenAI function calling
• Type-Safe Processing using Pydantic models
• 7-Step Content Pipeline: SEO, Marketing Brief, Article Generation, SEO Optimization, Internal Docs, Content Formatting, Design Kit
• Quality Scoring with confidence metrics for every step
• Human-in-the-Loop approval system for quality control
• Template-Based Prompts with Jinja2 for easy customization

🔒 Enterprise Security

• API Key Authentication with role-based access control
• Advanced Rate Limiting with attack detection and IP whitelisting
• Input Validation preventing SQL injection, XSS, and command injection
• Security Audit Logging with comprehensive event tracking
• Content Sanitization and validation

🚀 High Performance

• 20% Faster than agent-based approach
• 10% Lower Costs with optimized token usage
• Intelligent Caching with TTL and LRU eviction
• Connection Pooling for database optimization
• Real-time Performance Monitoring with metrics collection

🗄️ Database Support

• SQL Databases: SQLite, PostgreSQL, MySQL
• NoSQL Databases: MongoDB, Redis
• Content Sources: File, API, Web Scraping, Webhook
• Health Monitoring and connection management

🌐 Production Ready

• Docker & Kubernetes deployment with HPA and monitoring
• AWS CloudFormation templates for complete infrastructure deployment
• CI/CD Pipeline with security scanning and performance testing
• Comprehensive Testing with 1,350+ tests covering all critical paths
• API Documentation with Swagger/OpenAPI
• Monitoring & Observability with Prometheus and Grafana

🚀 Quick Start

Using pip (Recommended)

# Clone
git clone https://github.com/your-org/marketing-project.git
cd marketing-project

# Create virtual environment (recommended)
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# UDPipe English model is automatically downloaded on first use
# If you need to download it manually, it will be cached at ~/.udpipe/models/

# For development (optional)
pip install -r requirements-dev.txt

# Install the project in development mode
pip install -e .

# Set up environment
cp env.example .env
# Fill .env with your secrets (especially OPENAI_API_KEY)
# Redis defaults work for local development (localhost:6379)

# Run the marketing project
python -m marketing_project.main run

# Start the API server
python -m marketing_project.main serve

# Run tests
pytest

# Run security and database tests
python test_security_and_database.py

# Run load tests
python run_load_test.py --url http://localhost:8000 --test basic

Using Docker (Development)

# Start all services (Redis, API, Worker)
cd deploy/docker
docker-compose -f docker-compose.dev.yml up -d

# Check service health
docker-compose -f docker-compose.dev.yml ps

# View logs
docker-compose -f docker-compose.dev.yml logs -f api
docker-compose -f docker-compose.dev.yml logs -f worker

# Stop services
docker-compose -f docker-compose.dev.yml down

Using Docker (Production)

# Start all services
cd deploy/docker
docker-compose up -d

# Check service health
docker-compose ps

# View logs
docker-compose logs -f api

# Stop services
docker-compose down

🔌 API Endpoints

The Marketing Project provides a comprehensive REST API with enterprise-grade security and performance monitoring.

Core Processing Endpoints

Method	Endpoint	Description	Authentication
POST	/api/v1/process/blog	Process blog posts through AI pipeline	API Key
POST	/api/v1/process/transcript	Process transcripts (podcasts, videos)	API Key
POST	/api/v1/process/release-notes	Process software release notes	API Key
POST	/api/v1/analyze	Analyze content for marketing insights	API Key
POST	/api/v1/pipeline	Run complete pipeline with auto-routing	API Key

Content Source Endpoints

Method	Endpoint	Description	Authentication
GET	/api/v1/content-sources	List all content sources	API Key
GET	/api/v1/content-sources/{name}/status	Get source status	API Key
POST	/api/v1/content-sources/{name}/fetch	Fetch content from source	API Key

Job Management Endpoints

Method	Endpoint	Description	Authentication
GET	/api/v1/jobs/{job_id}/status	Get job status and progress	API Key
GET	/api/v1/jobs/{job_id}/result	Get completed job results	API Key
GET	/api/v1/jobs	List all jobs with filtering	API Key

Onboarding Examples Endpoints

Method	Endpoint	Description	Authentication
GET	/api/v1/onboarding-examples	List active quick-start job templates	JWT
GET	/api/v1/onboarding-examples/admin	List all templates including inactive (admin)	JWT (admin)
POST	/api/v1/onboarding-examples/admin	Create a new template (admin)	JWT (admin)
GET	/api/v1/onboarding-examples/admin/{id}	Get single template by ID (admin)	JWT (admin)
PATCH	/api/v1/onboarding-examples/admin/{id}	Update a template (admin)	JWT (admin)
DELETE	/api/v1/onboarding-examples/admin/{id}	Delete a template (admin)	JWT (admin)

Analytics & Quality Endpoints

Method	Endpoint	Description	Authentication
GET	/api/v1/analytics/dashboard	Dashboard statistics (admin)	JWT (admin)
GET	/api/v1/analytics/quality-scores	Per-job quality score history with pagination	JWT (admin)
GET	/api/v1/analytics/monitoring/unified	Unified monitoring metrics	JWT (admin)
GET	/api/v1/analytics/monitoring/cost	Cost tracking metrics	JWT (admin)
GET	/api/v1/analytics/monitoring/quality	Quality trend analysis	JWT (admin)

Health & Monitoring Endpoints

Method	Endpoint	Description	Authentication
GET	/api/v1/health	Health check for Kubernetes probes	None
GET	/api/v1/ready	Readiness check for Kubernetes probes	None
GET	/api/v1/performance/summary	Get performance summary and metrics	JWT
GET	/api/v1/security/audit	Get security audit logs	JWT
GET	/api/v1/cache/stats	Get cache statistics	JWT

Authentication

The application uses Keycloak for authentication and authorization. All protected endpoints require a valid JWT token from Keycloak.

Getting an Access Token

1. Authenticate with Keycloak through the frontend application, or
2. Use Keycloak's token endpoint directly:

curl -X POST "https://your-keycloak-server.com/realms/your-realm/protocol/openid-connect/token" \
     -H "Content-Type: application/x-www-form-urlencoded" \
     -d "client_id=your-client-id" \
     -d "client_secret=your-client-secret" \
     -d "grant_type=client_credentials" \
     -d "scope=openid"

Using the Access Token

Include the token in the Authorization header:

curl -H "Authorization: Bearer your-access-token-here" \
     -H "Content-Type: application/json" \
     -d '{"content": {"id": "test", "title": "Test", "content": "Test content", "snippet": "Summary"}}' \
     http://localhost:8000/api/v1/process/blog

Public Endpoints

The following endpoints do not require authentication:

• GET /api/v1/health - Health check
• GET /api/v1/ready - Readiness check

Role-Based Access Control

Some endpoints may require specific roles:

• Admin-only endpoints require the admin role
• Editor endpoints require the editor or admin role
• Regular user endpoints are accessible to all authenticated users

For detailed Keycloak setup instructions, see docs/KEYCLOAK_SETUP.md.

API Documentation

• Swagger UI: http://localhost:8000/docs
• ReDoc: http://localhost:8000/redoc
• OpenAPI Schema: http://localhost:8000/openapi.json

🤖 AI Function Pipeline

The core of this project is a sophisticated 7-step AI pipeline built on OpenAI's function calling feature.

Pipeline Architecture

Content Input
    ↓
Simplified Processor (blog, transcript, or release notes)
    ↓
FunctionPipeline (orchestrates 7 AI function calls)
    ↓
┌─────────────────────────────────────────┐
│ Step 1: SEO Keywords                    │
│ Step 2: Marketing Brief                 │
│ Step 3: Article Generation              │
│ Step 4: SEO Optimization                │
│ Step 5: Internal Documentation          │
│ Step 6: Content Formatting              │
│ Step 7: Design Kit & Visual Elements    │
└─────────────────────────────────────────┘
    ↓
Structured JSON Result (Pydantic models)

Key Benefits

✅ Guaranteed Structured Output - Every step returns typed JSON ✅ 20% Faster - No agent reasoning loops, direct function calls ✅ 10% Cheaper - Optimized token usage, no redundant processing ✅ Type-Safe - Pydantic models prevent runtime errors ✅ Quality Metrics - Confidence scores for every step ✅ Human Approval - Optional review before finalizing ✅ Template-Based - Easy to customize prompts via Jinja2

Example Usage

from marketing_project.processors import process_blog_post
import json

content = {
    "id": "blog-123",
    "title": "10 Marketing Tips",
    "content": "Here are 10 proven marketing strategies...",
    "snippet": "Top marketing tips for 2025",
    "author": "Jane Smith",
    "tags": ["marketing", "tips", "strategy"]
}

result_json = await process_blog_post(json.dumps(content))
result = json.loads(result_json)

# Access structured pipeline results
seo_keywords = result["pipeline_result"]["step_results"]["seo_keywords"]
marketing_brief = result["pipeline_result"]["step_results"]["marketing_brief"]
final_content = result["pipeline_result"]["final_content"]

🔒 Security Features

Authentication & Authorization

• API Key Authentication: Secure API key validation with role-based access
• Rate Limiting: Advanced rate limiting with IP and user-based limits
• Attack Detection: Automatic detection and blocking of suspicious patterns
• IP Whitelisting: Configurable IP whitelist for trusted sources

Input Validation

• SQL Injection Prevention: Pattern detection and query sanitization
• XSS Protection: Script injection detection and content sanitization
• Command Injection Prevention: Shell command pattern blocking
• Content Validation: Comprehensive input validation and sanitization

Security Monitoring

• Audit Logging: Comprehensive security event logging
• Risk Scoring: Automatic risk assessment for requests
• Anomaly Detection: Detection of unusual access patterns
• Security Alerts: Real-time security event notifications

🚀 Performance & Monitoring

Performance Monitoring

• Real-time Metrics: Request/response time, memory, CPU usage
• Performance Dashboards: Built-in performance monitoring
• Load Testing: Comprehensive load testing framework
• Query Optimization: Automatic database query optimization

Caching & Optimization

• Intelligent Caching: LRU cache with TTL support
• Connection Pooling: Database connection optimization
• Query Caching: Automatic query result caching
• Response Compression: Automatic response compression

Load Testing

# Basic load test
python run_load_test.py --url http://localhost:8000 --test basic

# Stress test
python run_load_test.py --url http://localhost:8000 --test stress

# Comprehensive test
python run_load_test.py --url http://localhost:8000 --test all

🐳 Deployment

Docker

# Build and run with Docker Compose
cd deploy/docker
docker-compose up -d

# Build production image
docker build -t marketing-project-api:latest -f deploy/docker/Dockerfile .

# Scale workers
docker-compose up -d --scale worker=4

Kubernetes

# Deploy to Kubernetes
kubectl apply -k deploy/k8s/

# Check deployment status
kubectl get pods -n marketing-project

# View logs
kubectl logs -f -n marketing-project deployment/marketing-project-api

AWS CloudFormation

# Set required environment variables
export OPENAI_API_KEY="your_openai_api_key_here"
export API_KEY="your_32_character_minimum_api_key_here"
export DATABASE_PASSWORD="your_database_password_here"
export MONGODB_PASSWORD="your_mongodb_password_here"

# Deploy to AWS
cd deploy/aws
./deploy-aws.sh -e production -r us-east-2

See deploy/AWS_DEPLOYMENT.md for detailed AWS deployment instructions.

Environment Configuration

Copy env.example to .env and configure:

Telemetry Configuration (Optional)

To enable telemetry tracing with Arthur, set the following environment variables:

• ARTHUR_BASE_URL: Base URL for Arthur API (default: http://localhost:3030)
• ARTHUR_API_KEY: API key for Arthur authentication (required for telemetry)
• ARTHUR_TASK_ID: Task ID for Arthur (required for telemetry, must have is_agentic=True)

If these variables are not set, the application will run normally without telemetry. Telemetry initialization is non-blocking and will not prevent the application from starting if misconfigured.

Other Environment Variables

• API keys and authentication
• Database connections
• Security settings
• Performance monitoring
• Logging configuration

🌐 Internationalization

Templates live under src/marketing_project/prompts/${TEMPLATE_VERSION}/{en,fr,...}/. Set TEMPLATE_VERSION=v1 in your .env.

To add a new language:

# Copy English templates
cp -r src/marketing_project/prompts/v1/en src/marketing_project/prompts/v1/es

# Translate the .j2 files
# Update your .env or environment
export LANG=es

📁 Project Structure

marketing-project/
├── src/marketing_project/     # Main application code
│   ├── api/                    # FastAPI routes and endpoints
│   ├── core/                   # Core models and utilities
│   ├── processors/             # Content processors (blog, transcript, release notes)
│   ├── services/               # Business logic and external integrations
│   │   └── function_pipeline.py  # AI function pipeline orchestrator
│   ├── prompts/                # Jinja2 templates for AI prompts
│   │   └── v1/en/              # English prompt templates
│   ├── middleware/             # FastAPI middleware (auth, logging, cors)
│   ├── models/                 # Pydantic data models
│   └── config/                 # Configuration management
├── tests/                      # Test suite (1,350+ tests)
│   ├── api/                    # API endpoint tests
│   ├── services/               # Service layer tests
│   └── plugins/                # Plugin tests
├── deploy/                     # Deployment configurations
│   ├── docker/                 # Docker and docker-compose files
│   ├── k8s/                    # Kubernetes manifests
│   └── aws/                    # AWS CloudFormation templates
├── docs/                       # Documentation
├── content/                    # Sample content for testing
└── requirements.txt            # Python dependencies

🧩 Architecture

This project follows a clean, simple architecture:

• Function-Based Pipeline - Direct OpenAI function calls with structured outputs
• Type-Safe Processing - Pydantic models for all data structures
• Template-Driven Prompts - Jinja2 templates for easy customization
• Multi-Locale Support - Internationalization ready
• Comprehensive Testing - pytest with async support
• Docker & K8s Ready - Production deployment ready
• Modern Python - Type hints, async/await, and best practices

Core Components

1. Processors (src/marketing_project/processors/)

   • Simple, focused processors for each content type
   • Validate input with Pydantic models
   • Call FunctionPipeline for AI processing
   • Return structured JSON results

2. FunctionPipeline (src/marketing_project/services/function_pipeline.py)

   • Orchestrates 7-step AI pipeline
   • Uses OpenAI function calling for guaranteed JSON
   • Loads prompts from Jinja2 templates
   • Returns typed Pydantic models

3. Templates (src/marketing_project/prompts/v1/en/)

   • Comprehensive 100-170 line prompts
   • Best practices and guidelines
   • Easy to customize and version

4. API Layer (src/marketing_project/api/)

   • FastAPI endpoints
   • Authentication and authorization
   • Job queue integration
   • Health checks and monitoring

📦 Dependency Management

# Install dependencies
pip install -r requirements.txt

# Install development dependencies
pip install -r requirements-dev.txt

# Add a new dependency
pip install package-name
# Then update requirements.txt manually

# Update dependencies
pip install --upgrade -r requirements.txt

# Show installed packages
pip list

# Freeze current environment
pip freeze > requirements-current.txt

🚀 Kubernetes Deployment

The project includes complete Kubernetes manifests for production deployment:

# Deploy to Kubernetes
kubectl apply -k deploy/k8s/

# Or deploy individual components
kubectl apply -f deploy/k8s/namespace.yml
kubectl apply -f deploy/k8s/configmap.yml
kubectl apply -f deploy/k8s/deployment.yml
kubectl apply -f deploy/k8s/service.yml
kubectl apply -f deploy/k8s/ingress.yml
kubectl apply -f deploy/k8s/hpa.yml
kubectl apply -f deploy/k8s/cronjob.yml

Features:

• Auto-scaling - HPA based on CPU and memory usage
• Health checks - Liveness and readiness probes
• TLS termination - Secure HTTPS access
• Resource limits - Prevents resource exhaustion
• Scheduled execution - CronJob for automated processing
• Monitoring ready - Metrics endpoint and structured logging

See deploy/k8s/README.md for detailed deployment instructions.

📚 Documentation

See docs/ for:

• Architecture diagrams
• API reference
• Deployment guides
• Development guidelines

🤝 Contributing

Please read CONTRIBUTING.md for development guidelines and best practices.

📝 License

This project is licensed under the MIT License. See LICENSE for details.

---

🎉 What's New

Recent Improvements (2025)

• ✅ Migration to Function Pipeline - 83% code reduction, 20% faster
• ✅ Quality Scoring - Confidence metrics for all pipeline steps
• ✅ Human-in-the-Loop - Approval system for quality control
• ✅ Template System - Easy prompt customization with Jinja2
• ✅ Simplified Architecture - Clean, maintainable codebase
• ✅ Comprehensive Documentation - 7,400+ lines of documentation

For migration details, see the documentation files in the project root.