Maintaining 99.96% uptime as a lending business doubled

Why were third-party systems such a big reliability problem?

KYC was a major source of friction.

The company depended on external KYC providers, lender APIs and mandate systems. Some of these were older state-owned systems too, so there was only so much the company could control from its side.

If a KYC provider went down, users could get stuck during onboarding.

The team first needed to know where the failure was actually coming from. Was it something inside the company, or was one of the providers having a problem? Even if it was a provider, users still couldn’t just be left stuck there.

How did we figure out where a failure was coming from?

We put Datadog APM across the lending services. KYC, mandates, lender integrations and disbursals were all covered.

The team could trace a request through the company’s services and see which third-party calls it made.

We also gave the external providers their own dashboards. So instead of looking at KYC as one big thing, the team could see what was happening with each vendor or lender API separately.

This made it much easier to tell whether the company’s service had a problem or whether the issue was with the provider.

We also started including observability in the acceptance criteria for new work. Teams had to decide what they wanted to monitor while they were building the feature instead of coming back and doing it later.

What changed for the on-call team?

The team was getting around 800 pages a week. On-call engineers had to work through that noise before they could identify which alerts actually needed attention.

We tied alerts back to SLOs and error budgets to cut some of that down. We connected Datadog to PagerDuty and the team had proper on-call rotations and escalation paths there.

We also split incidents by severity, from P0 to P3.

For the important alerts, we wrote runbooks. So, whoever got paged didn’t have to wait around for the person who knew that service best before they could do anything.

We also started doing blameless RCAs for P0 and P1 incidents. The team reviewed the follow-up work from those incidents each week. Blue-green deployments helped with releases too. If a release caused a problem, the team could switch traffic back quickly.

MTTD came down from 30 minutes to 3 minutes and P0 & P1 incidents went from around 8-10 a month to 2-3.

And those 800 weekly pages also came down to around 30.

How did we stop one problem from affecting others?

We used Resilience4J circuit breakers around the third-party calls.

If a provider started slowing down or failing, we didn’t want more requests to just keep piling up. The circuit breaker could stop sending requests there.

We also separated the resources used by each provider. Each provider also got its own thread pool and connection pool. So if one KYC vendor became slow, it couldn’t take up the capacity the others needed.

We set timeouts separately for each provider too. We were careful with retries too and only retried a request when it was safe to do so.

We also moved a lot of database reads to replicas so the primary database had less work to do as traffic grew.

What happened when a KYC provider or lender went down?

The company was already working with multiple KYC vendors and lenders. We wanted the system to be able to switch providers if one of them started having problems.

Each provider had its own adapter, but the company’s services could work with all of them through the same interface. We also added routing based on how the providers were performing at that moment.

If one provider started slowing down or failing more, traffic could be sent to another one. The circuit breaker was checked as well.

The team could change these routing rules without doing a new deployment. They could send more traffic to one provider, for example, or change which provider got priority. Cost and approval rate were important factors as well.

This made a significant difference during KYC outages.

Before, a provider going down could completely block users from finishing KYC. After the changes, around 90% of KYC journeys could still go through.

How did the system hold up through the growth?

The company maintained 99.96% uptime through this period, even when the lending business was growing as quickly as it was.

Onboarding failures came down by 40%. Repayment settlement got much faster too. It came down from hours to minutes.