Infrastructure Cost Optimization for AI Agents: Right-Sizing Compute and Storage

Infrastructure Costs Are the Silent Budget Killer

Teams obsess over LLM token costs while running oversized compute instances 24/7. For many AI agent deployments, infrastructure costs (compute, storage, networking) rival or exceed LLM API costs. A single m5.2xlarge instance running idle at night costs $277/month. Multiply that by a few services, add a vector database cluster, and infrastructure alone can hit $2,000–$5,000/month before you send a single API call.

The fix is systematic: measure actual resource usage, right-size instances, implement auto-scaling, and use pricing tiers (spot, reserved) strategically.

Measuring Resource Utilization

Before optimizing, you need to know what you are actually using.

import psutil
import time
from dataclasses import dataclass, field
from typing import List

@dataclass
class ResourceSnapshot:
    timestamp: float
    cpu_percent: float
    memory_percent: float
    memory_used_mb: float
    disk_used_percent: float
    network_bytes_sent: int
    network_bytes_recv: int

class ResourceMonitor:
    def __init__(self):
        self.snapshots: List[ResourceSnapshot] = []

    def capture(self) -> ResourceSnapshot:
        net = psutil.net_io_counters()
        snapshot = ResourceSnapshot(
            timestamp=time.time(),
            cpu_percent=psutil.cpu_percent(interval=1),
            memory_percent=psutil.virtual_memory().percent,
            memory_used_mb=psutil.virtual_memory().used / (1024 * 1024),
            disk_used_percent=psutil.disk_usage("/").percent,
            network_bytes_sent=net.bytes_sent,
            network_bytes_recv=net.bytes_recv,
        )
        self.snapshots.append(snapshot)
        return snapshot

    def utilization_summary(self) -> dict:
        if not self.snapshots:
            return {}
        return {
            "avg_cpu": round(sum(s.cpu_percent for s in self.snapshots) / len(self.snapshots), 1),
            "max_cpu": round(max(s.cpu_percent for s in self.snapshots), 1),
            "avg_memory": round(
                sum(s.memory_percent for s in self.snapshots) / len(self.snapshots), 1
            ),
            "max_memory": round(max(s.memory_percent for s in self.snapshots), 1),
            "p95_cpu": round(sorted(s.cpu_percent for s in self.snapshots)[
                int(len(self.snapshots) * 0.95)
            ], 1),
            "samples": len(self.snapshots),
        }

    def is_oversized(self) -> dict:
        summary = self.utilization_summary()
        return {
            "cpu_oversized": summary.get("p95_cpu", 0) < 30,
            "memory_oversized": summary.get("max_memory", 0) < 40,
            "recommendation": self._recommend(summary),
        }

    def _recommend(self, summary: dict) -> str:
        if summary.get("p95_cpu", 0) < 20 and summary.get("max_memory", 0) < 30:
            return "Strongly consider downsizing to a smaller instance"
        elif summary.get("p95_cpu", 0) < 40:
            return "Moderate opportunity to downsize"
        return "Current sizing appears appropriate"

Auto-Scaling Configuration

AI agent traffic follows predictable patterns: high during business hours, low at night. Auto-scaling matches capacity to demand.

from dataclasses import dataclass

@dataclass
class ScalingPolicy:
    min_replicas: int
    max_replicas: int
    target_cpu_percent: int
    target_memory_percent: int
    scale_up_cooldown_seconds: int = 60
    scale_down_cooldown_seconds: int = 300

ENVIRONMENT_POLICIES = {
    "production": ScalingPolicy(
        min_replicas=2,
        max_replicas=20,
        target_cpu_percent=60,
        target_memory_percent=70,
        scale_up_cooldown_seconds=30,
        scale_down_cooldown_seconds=300,
    ),
    "staging": ScalingPolicy(
        min_replicas=1,
        max_replicas=3,
        target_cpu_percent=70,
        target_memory_percent=80,
    ),
}

def generate_k8s_hpa(name: str, policy: ScalingPolicy) -> dict:
    return {
        "apiVersion": "autoscaling/v2",
        "kind": "HorizontalPodAutoscaler",
        "metadata": {"name": f"{name}-hpa"},
        "spec": {
            "scaleTargetRef": {
                "apiVersion": "apps/v1",
                "kind": "Deployment",
                "name": name,
            },
            "minReplicas": policy.min_replicas,
            "maxReplicas": policy.max_replicas,
            "metrics": [
                {
                    "type": "Resource",
                    "resource": {
                        "name": "cpu",
                        "target": {
                            "type": "Utilization",
                            "averageUtilization": policy.target_cpu_percent,
                        },
                    },
                },
            ],
            "behavior": {
                "scaleDown": {
                    "stabilizationWindowSeconds": policy.scale_down_cooldown_seconds,
                },
            },
        },
    }

Spot Instance Strategy

Spot instances offer 60–90% savings over on-demand pricing but can be interrupted. Use them for stateless, fault-tolerant agent workloads.

@dataclass
class SpotStrategy:
    on_demand_base: int  # minimum on-demand instances for reliability
    spot_ratio: float    # percentage of additional capacity to run on spot
    instance_types: List[str]  # diversify across types for availability
    fallback_to_on_demand: bool = True

RECOMMENDED_STRATEGIES = {
    "agent_workers": SpotStrategy(
        on_demand_base=2,
        spot_ratio=0.70,
        instance_types=["m5.large", "m5a.large", "m6i.large"],
    ),
    "batch_processors": SpotStrategy(
        on_demand_base=0,
        spot_ratio=1.0,
        instance_types=["c5.xlarge", "c5a.xlarge", "c6i.xlarge"],
    ),
    "vector_database": SpotStrategy(
        on_demand_base=3,
        spot_ratio=0.0,  # never use spot for stateful data stores
        instance_types=["r5.xlarge"],
    ),
}

Storage Optimization

AI agent systems generate large volumes of logs, traces, and conversation histories. Implement tiered storage with automatic lifecycle policies.

STORAGE_TIERS = {
    "hot": {
        "retention_days": 7,
        "storage_type": "SSD",
        "cost_per_gb_month": 0.10,
        "use_for": ["active conversations", "recent traces", "cache"],
    },
    "warm": {
        "retention_days": 90,
        "storage_type": "HDD / S3 Standard",
        "cost_per_gb_month": 0.023,
        "use_for": ["historical conversations", "analytics data"],
    },
    "cold": {
        "retention_days": 365,
        "storage_type": "S3 Glacier",
        "cost_per_gb_month": 0.004,
        "use_for": ["audit logs", "compliance archives"],
    },
}

FAQ

How do I decide between right-sizing and auto-scaling?

Do both. Right-size first to establish the correct baseline instance type, then add auto-scaling to handle demand fluctuations. Right-sizing without auto-scaling wastes money during off-peak hours. Auto-scaling on oversized instances scales the wrong resource — you end up adding more capacity than needed per replica.

Are spot instances safe for production AI agent workloads?

Yes, for stateless worker processes that can tolerate restarts. Run a base layer of on-demand instances (enough to handle minimum expected traffic) and use spot for burst capacity. Never use spot for stateful services like databases, vector stores, or in-memory caches that would lose data on termination.

How much can I realistically save with infrastructure optimization?

Teams that have never optimized typically find 30–50% savings from right-sizing alone. Adding auto-scaling saves another 15–25% on variable workloads. Spot instances for eligible workloads add another 20–30% savings on those specific instances. Combined, total infrastructure cost reductions of 40–60% are common.

---

#Infrastructure #CostOptimization #AutoScaling #CloudComputing #Kubernetes #AgenticAI #LearnAI #AIEngineering