How We Replaced Our Kubernetes Ingress Controller Before It Got Put Down

At Rover, we take care of pets. And apparently, deprecated infrastructure too.

We’re Mariana and Van, and we’re part of the SRE team at Rover. Our team’s job is to own the infrastructure layer (the AWS accounts, the Kubernetes clusters, the Terraform, the security posture, the CI/CD pipelines) so that product teams can focus on building instead of wrestling with platforms. We like to think of ourselves as the people who make the complicated stuff boring, in the best possible way.

This is the story of one of those projects.

Every engineer knows the feeling. You’re sipping your morning coffee, skimming release notes, and then you see it: “End of Life: March 2026.” Your stomach drops. That component? It’s the front door to every internal tool your company runs.

That’s exactly what happened to us when the NGINX Ingress Controller, the workhorse that had faithfully routed traffic to our internal Kubernetes services for years, announced its retirement. Like an aging golden retriever who’s given you the best years of its life, it was time to find a new approach.

Here’s the story of how we migrated to AWS Application Load Balancers with built-in JWT validation, wrote a 20-line Cloudflare (CF) Worker that saved the whole project, and came out the other side with a more secure, simpler architecture.

What the Old Setup Did

When Rover began migrating our application into Kubernetes, we needed a way to expose our services to the internet so that we can route traffic to the apps accordingly. At that time the suggested option by the community was to use NGINX Ingress Controller. Below is an architectural diagram of our NGINX Ingress Controller implementation:

 

1. Initial User Request comes in.

2. Request will hit the AWS ALB.

3. Request from ALB will go to NGINX Controller and then routed accordingly based on the Kubernetes services matching hostname and path.

4. K8s SVC will take incoming traffic and send the request to K8s Pods.

Our NGINX Ingress Controller became the single entry point for all externally-accessible internal tools: dashboards, admin panels, data platforms, CI tooling. It sat inside our Amazon EKS clusters and handled two critical jobs:

• Routing: Matching incoming requests to the right backend service based on hostname and path.

• Authentication: Verifying that the person making the request was actually allowed to be there, via JWT tokens from Cloudflare Access.

It worked. For years, it worked great. However one day we saw a x.com post from the DevOps community here, which spun us to check Kubernetes official blog to see the deprecation notes here, We did not know that the NGINX Ingress Controller project was only handled by two maintainers and kept running into security holes ultimately leading to the deprecation. Despite NGINX’s proven performance, its lack of active maintenance posed a significant security liability. We weren’t willing to bet our internal infrastructure on a project that was essentially end-of-life.

So as we learned that NGINX Ingress controller was being deprecated we started looking into alternative solutions many of which have become the “new” standard including Traefik, HAProxy, Envoy-based options. Each of these options were viable however we would be introducing risks with a new controller such as time to understand the new tool, how to maintain, and future risks of the tool being deprecated.

Choosing the Replacement: Why AWS ALB?

An option in replacing NGINX Ingress Controller was to leverage AWS’s native ALB solution. In our old configuration we used Cloudflare (CF) Access Proxy as a way to validate that ONLY Rover employees had access to our applications. CF Access Proxy uses a JWT token that was validated via a Kubernetes side car running alongside the NGINX Ingress Controller. This was a hard requirement of the new solution.

Luckily, AWS had shipped the ability to natively validate JWT on the ALB directly. This meant we could push both routing and authentication out of the cluster entirely. No more managing an ingress controller. No more worrying about its CVEs, upgrades, or resource consumption inside our EKS clusters.

AWS ALB became the clear winner for Rover as this solution is less reliant on 3rd party software, little maintenance required, and the teams existing familiarity with ALB.

The design solution for using ALB ran into new questions including:

• How many ALBs should we create?

• Should we be in the business of creating multiple ALBs for each of our services so that they are segmented?

• What are the cost and maintenance implications with running multiple ALBs?

Rover’s design process requires a formal Request for Comment (RFC) document, and within a month of spotting the deprecation notice we had gone from “we should look into this” to a completed RFC with a proposed replacement and migration plan. The March 2026 end-of-life date gave us a runway, but we wanted to stay ahead of the deadline.

The RFC was reviewed with the SRE team at Rover and ultimately was concluded that using AWS ALB for replacing NGINX Ingress controller was the best choice. Not only did the solution ensure the same functionality as we had with NGINX Ingress Controller while offloading any future maintenance as well familiarity with ALBs made this design favorable. The team decided that one ALB was enough as having multiple ALBs introduces more churn on maintenance and incurs a higher cost as AWS charges by per ALB. By using just one ALB we can simply add routing rules here and route to a per service basis in Kubernetes.

The move also improved our security posture in ways we didn’t initially expect. With NGINX, authentication lived inside the cluster as the sidecar container we mentioned earlier. With ALB, we pushed authentication entirely outside the cluster boundary. We configured the ALB’s security group to only accept traffic from Cloudflare’s published IP ranges, meaning you can’t even reach the ALB without going through Cloudflare’s DDoS protection and WAF first. We enforced TLS 1.3 with no option to negotiate down. And because each application in Cloudflare Access gets its own audience identifier, the ALB doesn’t just check “is this a valid JWT?” but “is this JWT for this specific app?” A compromised token for one dashboard can’t unlock another.

From Ingress Controller to ALB: The New Architecture

The new setup is refreshingly simple:

Instead of traffic flowing through an in-cluster proxy that needed its own deployment, scaling, and monitoring, requests now go straight from Cloudflare to an AWS ALB that validates the JWT before forwarding to the target pod. The ALB is a fully managed service: no pods to babysit, no OOM kills at 3 AM.

Route configuration became declarative. Each service gets a block of config that says: “For this hostname and these paths, require a valid JWT with this audience claim, and send traffic to this Kubernetes service.” Adding a new authenticated service went from a multi-file YAML adventure to a single, readable config block.

However the new architecture had one gap to close. Cloudflare Access authenticates users and attaches a JWT to every request, but it places the token in a custom header (Cf-Access-Jwt-Assertion). AWS ALB’s native JWT validation expects tokens in the standard Authorization: Bearer <token> header. Two managed services, one header mismatch.

We solved this with a Cloudflare Worker, a lightweight serverless function running at Cloudflare’s edge, sitting between Access and the ALB. Its only job: read the JWT from Cf-Access-Jwt-Assertion and copy it into the Authorization header before the request leaves Cloudflare’s network. About 20 lines of JavaScript.

We also built in a guard: if a request arrives with both headers already set (a potential token injection attempt), the Worker returns a 409 Conflict rather than silently choosing one.

The Migration: One Service at a Time

We didn’t flip a switch. These services are the internal tools that Rover engineers rely on every day: dashboards, admin panels, CI tooling. If authentication broke during cutover, people would get locked out. If JWT validation on the ALB didn’t behave exactly like the old sidecar, requests could fail silently. DNS propagation added uncertainty to every switch. And we needed to coordinate with the teams depending on these tools so nobody was surprised mid-migration.

At Rover, all infrastructure is managed as code through Terraform, which gave us the safety net we needed. Every change was a pull request, reviewed by the team, and reversible. The migration followed a deliberate sequence:

1. Set up the new ALB with the Cloudflare-only security group and JWT validation rules.

2. Deploy the Cloudflare Worker to handle the header transformation.

3. Migrate one service: update its DNS to point to the new ALB, verify authentication and routing, confirm with the owning team.

4. Repeat for the next service.

Once all services were migrated, remove the old ingress controller entirely.

Each cutover was a small, reversible change. If something went wrong, we could point DNS back to the old ingress controller in minutes. Moving one service at a time meant we could catch issues early, before they affected everything, and we never had an outage. The whole migration, first service to last, took about two weeks.

What We Learned

Managed services are worth the trade-off. Running your own ingress controller gives you maximum flexibility, but also maximum responsibility. For our use case (internal tools with Cloudflare Access auth), an AWS ALB with native JWT validation was strictly better. Less code to maintain, fewer things to break.

Cloudflare Workers are underrated. That tiny header-transformation Worker solved what could have been a major architectural headache. When two systems disagree on a convention (Cloudflare Access’s Cf-Access-Jwt-Assertion vs. ALB’s Authorization: Bearer), a lightweight shim at the edge is often cleaner than trying to change either system.

Deprecation deadlines are gifts. It’s easy to see an end-of-life announcement as a burden. But it forced us to re-evaluate our architecture with fresh eyes, and we ended up with something simpler and more secure than what we started with. Sometimes the best refactors are the ones you’re forced into.

Version Tracking Runbook. A new procedure on the SRE team at Rover is to maintain a doc that outlines all of the packages owned by the team, current installed version, and when the version was updated. This check list acts as a source of truth for the team to audit when we should upgrade the packages accordingly. In addition we have documented a run book as to how we can update these packages when the time comes so that any SRE engineer can perform the upgrade.

Migrate incrementally. One service at a time, with a rollback plan for each. It’s slower than a big bang migration, but you sleep better.

Wrapping Up

From the first deprecation alert to the old ingress controller being fully decommissioned, the entire project took about a month. At Rover, our mission is connecting pet parents with the perfect pet care providers. That means our platform needs to be reliable, secure, and (let’s be honest) not running on deprecated infrastructure. This migration was one of those behind-the-scenes projects that no user will ever notice, and that’s exactly the point. The best infrastructure work is invisible.

The NGINX Ingress Controller served us well. But like teaching an old dog new tricks, sometimes it’s better to welcome a new member to the family, one that’s AWS-managed, Cloudflare-protected, and doesn’t need us to manage its pods.

Good boy, NGINX. You earned your retirement.