SwiftDeploy: OPAを用いた可観測なポリシー駆動型デプロイメントエンジンの構築

Introduction As part of the HNG Internship DevOps Track Stage 4B, I extended my Stage 4A project — SwiftDeploy — into a fully observable, policy-aware deployment platform. In Stage 4A, SwiftDeploy could: generate infrastructure files from a declarative manifest deploy containers using Docker Compose manage deployment modes (stable/canary) configure Nginx automatically Stage 4B transformed it into something much closer to a real production deployment system by adding: Prometheus instrumentation Open Policy Agent (OPA) policy enforcement live operational dashboards deployment safety gates audit logging and reporting chaos engineering validation The result is a deployment tool that not only deploys services, but also decides whether deployments are safe enough to proceed. The Core Philosophy: One Manifest, Everything Else Generated SwiftDeploy is built around a single principle: manifest.yaml is the only file you should ever edit manually. Everything else is generated from it. Here is the manifest structure: services: name: app image: swift-deploy-1-node:latest port: 3000 version: "1.0.0" mode: stablenginx: image: nginx:latest port: 8080 proxy_timeout: 30network: name: swiftdeploy-net driver_type: bridge From this manifest, the CLI generates: generated/nginx.conf generated/docker-compose.yml OPA runtime configuration This design provides: consistency reproducibility environment portability infrastructure-as-code discipline The grader can delete all generated files and rerun: ./swiftdeploy init and the entire stack regenerates correctly. Architecture Overview The system architecture consists of four major components: User ↓Nginx Reverse Proxy ↓Flask API Service ↓Prometheus Metrics ↓SwiftDeploy CLI ↓OPA Policy Engine The deployment stack includes: Flask application container Nginx reverse proxy Open Policy Agent (OPA) internal Docker network named log volumes The SwiftDeploy CLI The heart of the project is the swiftdeploy executable. It is a Python-based CLI tool that manages the entire deployment lifecycle. Supported Commands CommandPurposeinitGenerate config files from templatesvalidateRun pre-flight validation checksdeployStart the stackpromote canarySwitch deployment into canary modepromote stableReturn deployment to stable modestatusLive metrics dashboardauditGenerate audit reportteardownDestroy containers and networks The API Service The API service is a Flask application that supports both stable and canary deployment modes. Deployment mode is controlled through the MODE environment variable. Endpoints Root Endpoint GET / Returns: deployment mode version timestamp Example: { "message": "Welcome to SwiftDeploy", "mode": "stable", "version": "1.0.0"} Health Endpoint GET /healthz Returns: health status application uptime Chaos Endpoint POST /chaos Available only in canary mode. Supports: { "mode": "slow", "duration": 3 } { "mode": "error", "rate": 0.5 } { "mode": "recover" } This endpoint was used to simulate: degraded latency random failures recovery workflows Instrumentation: The /metrics Endpoint One of the biggest upgrades in Stage 4B was observability. I instrumented the Flask service using the prometheus_client library. The service now exposes: GET /metrics in Prometheus text format. Metrics Collected Request Throughput http_requests_total Labels: method path status_code Example: http_requests_total{method="GET",path="/",status_code="200"} 152 Request Latency http_request_duration_seconds Histogram used for: latency analysis P99 calculation Application Uptime app_uptime_seconds Tracks process uptime. Deployment Mode app_mode Values: 0 = stable 1 = canary Chaos State chaos_active Values: 0 = none 1 = slow 2 = error Why Metrics Matter Without metrics: deployments are blind failures become invisible canary safety cannot be enforced Metrics became the foundation for: policy decisions dashboards auditing promotion safety Open Policy Agent (OPA): The Brain of SwiftDeploy The most important design principle in Stage 4B was: The CLI must never make allow/deny decisions itself. All decision-making lives entirely inside OPA. SwiftDeploy only: gathers data sends context to OPA acts on the response This separation makes the system: modular secure maintainable extensible OPA Policy Domains I separated policies into independent domains. Each policy: answers one question owns its own logic operates independently Infrastructure Policy Runs before deployment. Blocks deployment when: disk free space is below 10GB CPU load exceeds 2.0 Rego Example package infradefault allow = falseallow { input.disk_free_gb >= data.thresholds.disk_free_gb input.cpu_load <= data.thresholds.cpu_load} Canary Safety Policy Runs before promotion. Blocks promotion when: error rate exceeds 1% P99 latency exceeds 500ms Rego Example package canarydefault allow = falseallow { input.error_rate <= data.thresholds.error_rate input.p99_latency_ms <= data.thresholds.p99_latency_ms} Policy Thresholds Thresholds are stored separately in: policies/data.json Example: { "thresholds": { "disk_free_gb": 10, "cpu_load": 2.0, "error_rate": 0.01, "p99_latency_ms": 500 }} This prevents: hardcoded values duplicated configuration policy coupling OPA Isolation The OPA container runs on an internal Docker network. It is intentionally NOT exposed through Nginx. Only the CLI can access OPA directly via: http://localhost:8181 This prevents external users from: querying policies bypassing deployment logic inspecting internal rules This mirrors real production security architecture. Pre-Deploy Policy Enforcement Before deployment, SwiftDeploy collects: CPU load available disk space Example payload: { "disk_free_gb": 8.5, "cpu_load": 2.4} OPA evaluates the payload. If policies fail: Deployment blocked:Infrastructure policy violation The deployment never proceeds. Canary Safety Enforcement Before promotion, SwiftDeploy: scrapes /metrics calculates error rate calculates P99 latency submits metrics to OPA If the canary is unhealthy: promotion is blocked rollout is prevented This introduces production-grade deployment safety. The Status Dashboard The status command provides a live operational dashboard. ./swiftdeploy status The dashboard: refreshes continuously scrapes live metrics calculates request rate calculates P99 latency evaluates policy compliance appends results to history.jsonl Example output: SwiftDeploy Status Dashboard==================================================Mode: canaryChaos: errorError Rate: 52%P99 Latency: 430msPolicy Compliance:✓ Infrastructure policy: PASSING✗ Canary safety policy: FAILING Chaos Engineering This was one of the most interesting parts of the project. I intentionally injected: high error rates slow responses Example: curl -X POST http://localhost:8080/chaos -d '{"mode":"error","rate":0.9}' Immediately: metrics reflected failures policies began failing promotions were blocked This validated that: metrics were accurate policies were functional safety gates worked correctly Audit Logging Every: deploy promote status scrape policy violation is appended to: history.jsonl Example entry: { "timestamp": "2026-05-06T12:00:00", "mode": "canary", "error_rate": 0.52} Audit Report Generation Running: ./swiftdeploy audit generates: audit_report.md The report includes: deployment timeline mode changes chaos injections policy violations Example: | Timestamp | Policy | Details ||-----------|--------|---------|| 2026-05-06T00:47:10Z | Canary Safety | error_rate=50% | Challenges Faced a. Python Virtual Environment Issues Ubuntu’s externally-managed Python environment caused repeated package installation failures. The solution was: recreating the virtual environment installing dependencies inside the venv only b. Nginx Validation Problems Generated Nginx configs initially failed validation due to unresolved upstream references. Fix: validate only inside container context avoid host-side upstream resolution c. Metrics Parsing Calculating: error rate P99 latency from Prometheus text format required careful parsing and aggregation. d. OPA Failure Handling The CLI had to gracefully handle: OPA downtime connection failures malformed responses The system never crashes when OPA becomes unavailable. Lessons Learned Declarative Systems Scale Better A single source of truth drastically reduces configuration drift. Observability Is Mandatory Without metrics: policy enforcement becomes impossible deployments become blind Policy Engines Should Be Isolated Keeping OPA internal-only mirrors real enterprise architectures. Chaos Engineering Builds Confidence Breaking the system intentionally proved that: metrics were accurate policies were effective safety mechanisms worked Automation Must Be Explainable Every policy response included human-readable reasoning. This made debugging and operational decisions much easier. Final Thoughts Stage 4B transformed SwiftDeploy from a deployment generator into a lightweight deployment platform with: observability governance auditing deployment safety The project demonstrated how: metrics policy engines infrastructure generation deployment orchestration can work together to create reliable deployment systems. Most importantly, it reinforced a key DevOps principle: Safe automation is more valuable than fast automation. Introduction As part of the HNG Internship DevOps Track Stage 4B, I extended my Stage 4A project — SwiftDeploy — into a fully observable, policy-aware deployment platform. In Stage 4A, SwiftDeploy could: generate infrastructure files from a declarative manifest deploy containers using Docker Compose manage deployment modes (stable/canary) configure Nginx automatically Stage 4B transformed it into something much closer to a real production deployment system by adding: Prometheus instrumentation Open Policy Agent (OPA) policy enforcement live operational dashboards deployment safety gates audit logging and reporting chaos engineering validation The result is a deployment tool that not only deploys services, but also decides wh