Cloud Cost Optimisation: Where the Money Actually Goes

Where does cloud spend actually accumulate?

Every cloud cost optimisation exercise starts the same way: export three months of billing data at resource level and sort descending. The result is almost always steeply skewed. A handful of resources — a database cluster, one instance family, a NAT gateway, a logging pipeline — account for the bulk of the bill, and a long tail accounts for the rest. Production runs on a few large, always-on things; the tail is experiments, forgotten environments and defaults nobody revisited.

This is why the first audit is disproportionately productive and the fifth is not. The first pass works the top ten line items and finds real money in most of them. Fifty resources at twenty dollars a month take as long to investigate and return a fraction of the saving. Work the head of the distribution, automate the tail through policy, and stop.

Compute you are paying for and are not using

The largest recurring waste is compute sized for a peak that never comes. An instance chosen during a launch, or copied from the last project, sits at single-digit CPU utilisation for two years because nobody was asked to check. Right-sizing is only credible when it comes from measurement: take the p95 of CPU and memory over at least a fortnight, keep headroom for your real traffic pattern, and resize. Guessing downward is how you get a rollback and a team that never tries again.

Kubernetes hides the same problem behind a layer. The scheduler places pods using resource requests rather than actual usage, so a fleet with generous requests runs nodes at low utilisation while reporting itself as full. Set requests from observed usage — the Vertical Pod Autoscaler in recommendation mode is a reasonable source — and stop pinning limits to requests out of habit.

Non-production environments are the other reliable win, because they are sized like production and used during office hours. Staging, QA and demo each running continuously pay for 168 hours a week to be useful for perhaps fifty. A scheduled stop overnight and at weekends removes most of that without touching an application setting. The objection is always that someone might need it at 2am; the answer is a self-service start button.

• Instances left running after a load test, a cutover or a proof of concept — check anything over 90 days old with no recent deployment.
• Unattached block storage volumes, which bill at the full provisioned rate whether or not anything reads them.
• Snapshot chains with no retention policy, accumulating for years because each looks cheap in isolation.
• Old machine images and registry layers, especially where CI pushes an image on every commit.
• Public IPv4 addresses, which AWS now charges for hourly whether attached or not.
• Development clusters outliving the project, still paying a control plane charge.

Why data transfer charges surprise everyone

Nobody models data transfer during design, because in a data centre the network is a fixed cost and in a cloud it is a meter. Three charges do most of the damage. Traffic between availability zones inside one region is billed in both directions, so a service chatting to a database in another zone pays twice for the same bytes. Egress to the internet is billed per gigabyte on a tiered scale after a small free allowance. And a NAT gateway is charged by the hour and per gigabyte processed, so every private-subnet workload pulling images or calling an external API pays a toll on traffic it assumes is free. Check current published pricing before modelling any of this — rate cards move.

So a chatty architecture converts service decomposition straight into a data transfer line item. A request fanning out across six services spread over three zones can cross a zone boundary a dozen times, at both ends, before it returns to the user. Zone-aware routing, keeping caches and databases in the same zone as their heaviest readers, and VPC endpoints so traffic to managed services stays off the NAT gateway address most of it. For bulk egress — media, exports, model artefacts — a CDN in front of the origin is normally cheaper per gigabyte than serving directly.

Managed services, idle load balancers and observability spend

Managed services are where a size-too-large decision compounds: the tier bills continuously and is rarely revisited. A managed database running multi-AZ costs roughly double the single-instance price, correct for production and hard to justify for the QA copy nobody would miss for an hour. Storage autoscaling grows and never shrinks, so one runaway log table permanently raises the floor of the bill. Provisioned IOPS bought during an incident stay long after the query that caused it was fixed, and search and cache clusters are usually sized for a projected data volume rather than a reached one.

Load balancers are individually cheap and collectively invisible. Each application load balancer bills an hourly charge regardless of traffic, plus a capacity-unit charge. One is a rounding error; thirty — one per environment per service, created by a template and never removed — is a standing monthly cost for infrastructure that often serves nothing. Consolidating onto shared listeners with host or path routing removes most of them.

Observability is the fastest-growing line on most modern bills, and it grows faster than traffic rather than in step with it. Log platforms typically charge for ingestion, indexing and retention separately, so volume hits the bill up to three times. Debug logging left on after an incident, a health check logging every probe, and an architecture emitting a line per hop between services all push volume up without changing user-facing traffic. Sample high-volume traces, route audit logs to object storage rather than an indexed store, set retention per service, and check whether you are paying to index fields nobody queries.

Cost category, typical cause, and the fix that works

Structural levers, and where committed-use discounts trap you

Autoscaling that scales down

Most autoscaling groups scale up quickly and scale down never, because a nervous scale-down policy caused an incident once. That is a configuration for peak-cost operation. Scale down on a longer cooldown than you scale up, use a request or queue-depth signal where the workload is I/O-bound, and set the minimum to what the quiet period needs rather than to whatever makes the graph look reassuring.

Commitments, and the shape of the trap

AWS publishes discounts of up to around 72% for three-year all-upfront reserved instances and up to around 66% for Compute Savings Plans, with equivalent programmes on Azure and Google Cloud. The saving is genuine. The trap is that a three-year commitment bets your architecture will not change, and three years is long enough for a move to containers or a newer instance family to strand a chunk of it. Cover only your measured floor, prefer flexible commitment types, and ladder terms so they do not all expire in one quarter. Committing to a bloated baseline locks in the waste: right-size first, commit second.

Spot capacity for the right workloads

Spot capacity is heavily discounted and can be reclaimed on short notice — two minutes on AWS, around thirty seconds elsewhere. That suits CI runners, batch processing, media encoding, checkpointed training and stateless workers behind a queue. It does not suit anything holding session state in memory or a single-replica database. Diversify across instance types and zones so one pool being reclaimed cannot take the whole fleet, and handle the interruption notice rather than assuming it will not fire.

Storage lifecycle instead of manual tidying

Object storage tiering pays off as a policy, not as a periodic clean-up. Note the constraints first: infrequent-access tiers carry minimum storage durations and per-object minimum billable sizes, and archive tiers add retrieval fees and latency measured in hours. Millions of small objects can cost more in an IA tier than in standard once minimums apply, and intelligent tiering charges a per-object monitoring fee at high object counts. A lifecycle rule is a few lines and removes the decision permanently:

{
  "Rules": [
    {
      "ID": "logs-tiering",
      "Filter": { "Prefix": "logs/" },
      "Status": "Enabled",
      "Transitions": [
        { "Days": 30, "StorageClass": "STANDARD_IA" },
        { "Days": 120, "StorageClass": "GLACIER" }
      ],
      "Expiration": { "Days": 730 },
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 }
    }
  ]
}

That last clause earns its place: failed multipart uploads leave partial data that is billed as storage but does not appear in a normal object listing, so on a busy bucket it accumulates unnoticed for years.

Why nothing is fixable until cost is attributable

Cloud waste persists not because engineers are careless but because the bill arrives as one number addressed to nobody. Until each resource carries an owner, an environment and a service tag, no team can see its own spend, and a cost that belongs to everyone is optimised by no one.

Tagging has a detail that catches people out: on AWS, cost allocation tags must be activated in the billing console before they appear in cost reports, and activation is not retrospective — untagged history stays untagged. Turn them on before the audit. Enforce required tags at creation time through infrastructure-as-code policy rather than a wiki page, keep the vocabulary short (owner, environment, service, cost-centre), and accept that four enforced tags beat twelve aspirational.

With attribution in place, showback does most of the remaining work. A monthly figure per team, visible to that team, changes behaviour faster than any central mandate, because the engineers who see the number are the only people who know which resources are genuinely needed.

How to run the first cloud cost optimisation pass

1. Activate cost allocation tags and enable resource-level billing export first, so next month’s data is usable.
2. Sort three months of spend by resource and take the top twenty lines. Everything below that waits.
3. Delete first — orphaned volumes, dead snapshots, idle load balancers, unused IPs, finished environments. Deletion needs no planning.
4. Right-size from measured p95 utilisation, and schedule non-production off outside working hours.
5. Fix the structural charges: zone-aware placement, VPC endpoints, CDN for egress, trace sampling, log retention.
6. Only now buy commitments, against the floor you have just lowered, laddered across terms.
7. Set budget alerts and a per-service anomaly alert so the next regression surfaces in days, not in the next invoice.

Sequence matters more than technique. Every step except the last two makes the baseline smaller, and commitments bought against a fat baseline lock the fat in for years. Reverse the order and you book an impressive saving in month one, then spend three years paying for infrastructure you had already decided to remove.

If your bill has grown faster than your traffic and nobody can say precisely why, that is a solvable problem and usually a quick one to diagnose. We are glad to walk through a cloud cost optimisation review with your team and hand back the findings with the reasoning attached — talk to ApexStack about it.