Daniel Hidalgo's blog

One AI Setup, Two Worlds: How I Use AI Agents as a Tech Lead and as an Entrepreneur

Fri, 23 Oct 2026 00:00:00 GMT

I wrote before about what actually works with AI in the developer workflow. That post was about principles. This one is about the setup itself: the tools that run on my machine every day, how the same setup serves two very different contexts, and the rules I follow so that it stays a tool and not a liability.

The two contexts are my day job as a Tech Lead and Arupo, the company I'm building as a partner, plus a handful of personal projects. Very different constraints, one engineer, one laptop.

The Foundation: Configure Once, Use Everywhere

The biggest mistake I see is treating every AI tool as a separate island: one set of instructions in one editor, different habits in another, nothing shared between them. I went the other way. There is one foundation, and every agent sits on top of it.

A shared harness. I use Gentle AI to give every agent the same workflow, the same persona and the same review rules. It's configured for Claude Code, Codex, OpenCode, Gemini and Copilot. Whichever agent I open, it works the same way.
Persistent memory. Engram stores decisions, bug fixes, discoveries and preferences across sessions and across agents. When I come back to a project after two weeks, the agent doesn't start from zero, and neither do I.
A code map. Codegraph indexes each repository so an agent can understand the structure without reading file by file. That matters when the codebase is large and the context window is not.
Current documentation. Context7 gives agents the up-to-date docs of the libraries I use, instead of whatever version they memorized during training.
A second AI terminal. Pi, with the Gentle Shell, is where I run longer orchestrated work, with profiles for different models.
A guardian on commits. A pre-commit reviewer checks changes before they land.
A comfortable environment. Ghostty, Starship and the Gentleman dotfiles. It sounds cosmetic, but you spend the whole day there.

None of these is magic on its own. The value is that they are wired together: the same rules, the same memory, the same way of working, regardless of the agent or the project.

Two Worlds on One Machine

Here is the part people rarely talk about. My job and my company must never mix. Different accounts, different identities, different rules about what can leave the machine.

So the separation is automatic, by folder. When I enter my personal projects folder, the personal AI profile and my personal git and SSH identity activate on their own. When I enter the work folder, the corporate AI profile and the corporate git identity take over. I don't have to remember to switch, and that is the point: a rule you have to remember is a rule you will eventually break.

It isn't perfect, and I know where the gaps are. The GitHub CLI, for example, doesn't switch accounts per folder; it uses one global session. Knowing the gaps of your own setup is part of owning it.

At Work: Leading Inside the Constraints

A company has rules about tools and data, and they exist for good reasons. My rule is simple: the way of working travels with me, the company's policies decide which tools I can use. I adapt the setup to fit those policies, never the other way around.

But the bigger change at work isn't technical. It's how I spend my time as a Tech Lead. Agents take a lot of the mechanical work: exploring an unfamiliar service, drafting a migration, writing the boring tests. That gives me more time for what a Tech Lead is actually for: design decisions, code reviews, unblocking the team, incidents and postmortems.

One thing does not change: ownership stays human. An agent can propose a change to a production service. I'm the one who answers for it at 3 a.m.

At Arupo: Speed Without Lowering the Bar

In my own company the constraint is the opposite. There are no corporate restrictions, but there is very little time and a very small team. AI is what lets a small team ship like a bigger one, as long as the quality bar doesn't drop.

The best example is the website you're reading. In a few work sessions it went through:

A full redesign with English and Spanish versions.
An SEO cleanup.
Security hardening: a Content Security Policy generated at build time, with a check that fails the build if an inline script is missing from it.
A migration from Astro 5 to Astro 7.
An accessibility pass to meet WCAG AA contrast, and a dependency audit taken down to zero known vulnerabilities.

Each of those shipped as its own pull request, with preview deploys, before reaching production. The agents did most of the typing. I made every decision that mattered: what to upgrade and what to leave alone, which security trade-off to accept, when something was ready to merge.

The same setup runs my other projects: finance and fitness side products, a thesis, and the different repositories behind Arupo.

The Workflow That Ties It Together

Tools are the easy part. What makes this work is the process I follow with them, the same in both worlds:

Clarify first. If the request is ambiguous, the agent asks before it writes a single line.
Size the task. Small, well-understood changes are done directly. Large ones get tracked.
Write the spec. A large change gets a document: the requirements quoted exactly as I said them, a numbered task list and a log of decisions. Anyone, human or agent, can resume from it.
Tests first. When the behavior is testable, the failing test comes before the fix.
Small commits. Reviewable slices, never one giant diff that nobody can really review.
Review by risk. When a change touches security, data, contracts or deployments, an independent reviewer looks at it. When it doesn't, I don't pay for ceremony I don't need.
I merge. Pushing, opening the pull request and merging remain my decisions. Always.

What Doesn't Work (Yet)

To keep this honest:

Tools break. Some integrations need re-authentication, others fail to connect. A setup this connected needs maintenance.
Memory goes stale. A decision saved three months ago may no longer be true. Memory is context, not truth.
Agents are confidently wrong. They will tell you a build passes when it doesn't, or invent an API. That's why every claim needs evidence: the test output, the build log, the diff.

Closing Thoughts

AI didn't make me a different engineer. It amplified the one I already was. The fundamentals I learned over eight years, from network infrastructure to leading product teams, are exactly what let me direct these agents and catch them when they're wrong.

The skill that matters now isn't typing fast. It's knowing what to ask, recognizing when the answer is wrong, and owning the result. That part is still, and will remain, ours.

From Monolith to BFF: How We Actually Migrated Without Burning Everything Down

Sun, 15 Mar 2026 00:00:00 GMT

Let me tell you something that every senior engineer eventually learns: nobody wakes up one morning and decides to migrate away from a monolith for fun. You do it because the monolith is actively hurting your team, your deployments, and your customers. That's exactly where we were about two years ago, and I want to walk you through how we actually pulled it off — not the sanitized conference talk version, but the real one.

The Monolith Was Screaming for Help

Our monolith was a Node.js Express application that had been growing for three years. It started clean. It didn't stay that way. Here's what the symptoms looked like:

Deploy fear: A change to the payment module required deploying the entire application. One Friday afternoon, a CSS fix in the admin panel brought down the checkout flow. That was a fun weekend.
Tangled dependencies: The order service directly imported the user repository. The notification module knew about inventory levels. Everything knew about everything.
Team bottlenecks: Three squads working on the same codebase meant constant merge conflicts and a deploy queue that sometimes backed up for days.
Performance ceiling: We couldn't scale the reporting module independently from the real-time checkout flow. Both lived in the same process, competing for the same resources.

The monolith wasn't just technical debt — it was organizational debt. And that's when you know it's time.

Why BFF (And Not Just "Microservices")

The knee-jerk reaction is "let's do microservices!" But microservices without a clear frontend strategy just moves the spaghetti from the backend to the API layer. We had three clients: a web SPA (Angular), a mobile app (React Native), and an internal admin dashboard. Each one needed different data shapes, different auth flows, and different performance characteristics.

That's where Backend for Frontend comes in. The idea is simple: each client gets its own lightweight backend that orchestrates calls to domain services and shapes the response for that specific client's needs.

The key insight was this — the BFF is not a domain service. It has zero business logic. It's an orchestration and transformation layer. The moment you put business rules in a BFF, you've just built another monolith with extra network hops.

Hexagonal Architecture in the Domain Services

For the actual domain services (orders, inventory, users, payments), we went with hexagonal architecture. Here's why: we wanted the domain logic to be completely decoupled from infrastructure concerns. Database? That's an adapter. Message queue? Adapter. External payment gateway? Adapter.

Here's the structure we used in NestJS:

src/
├── domain/
│   ├── entities/
│   │   └── order.entity.ts
│   ├── ports/
│   │   ├── order.repository.port.ts
│   │   └── payment-gateway.port.ts
│   └── use-cases/
│       └── create-order.use-case.ts
├── infrastructure/
│   ├── adapters/
│   │   ├── postgres-order.repository.ts
│   │   └── stripe-payment.gateway.ts
│   └── modules/
│       └── order.module.ts
└── application/
    └── bff/
        └── web-bff.controller.ts

The port definition is clean and infrastructure-agnostic:

// domain/ports/order.repository.port.ts
export interface OrderRepositoryPort {
  findById(id: string): Promise<Order | null>;
  save(order: Order): Promise<Order>;
  findByCustomer(customerId: string, pagination: PaginationParams): Promise<PaginatedResult<Order>>;
}

And the BFF controller orchestrates without owning logic:

// application/bff/web-bff.controller.ts
@Controller('web/orders')
export class WebBffController {
  constructor(
    private readonly createOrderUseCase: CreateOrderUseCase,
    private readonly getCustomerUseCase: GetCustomerUseCase,
  ) {}

  @Post()
  async createOrder(@Body() dto: CreateOrderDto, @Req() req: AuthRequest) {
    const order = await this.createOrderUseCase.execute({
      customerId: req.user.id,
      items: dto.items,
      shippingAddress: dto.shippingAddress,
    });

    // BFF responsibility: shape response for the web client
    return {
      orderId: order.id,
      status: order.status,
      estimatedDelivery: order.estimatedDelivery?.toISOString(),
      items: order.items.map(item => ({
        name: item.productName,
        qty: item.quantity,
        total: formatCurrency(item.totalPrice),
      })),
    };
  }
}

Notice: the BFF calls the use case, then transforms the response for the web client. The mobile BFF would return a different shape — maybe with less data to save bandwidth, or with image URLs optimized for mobile screens.

The Migration Strategy: Strangler Fig

We didn't do a big bang migration. That's how you burn everything down. We used the strangler fig pattern, and here's how it worked in practice:

API Gateway as the router: We put an API Gateway (Kong) in front of everything. Initially, 100% of traffic went to the monolith.
Extract one domain at a time: We started with the notification service because it was the most isolated and lowest risk. We built it as a standalone NestJS service with hexagonal architecture.
Dual-write period: For about two weeks, both the monolith and the new service processed notifications. We compared outputs. Found three bugs in the new service this way.
Route switch: Once confident, we updated the gateway to route notification traffic to the new service. The monolith code stayed in place but dormant.
Repeat: Orders was next, then inventory, then payments (the scariest one, saved for last).

The whole migration took about eight months. Not eight days, not eight weeks. Eight months.

The Mistakes We Made

I'd be lying if I said it went smoothly. Here are the mistakes that cost us real time:

Mistake 1: Shared database too long. We initially let the new order service read from the monolith's database. "We'll separate it later," we said. Later took four months, and during that time we had schema coupling that defeated half the purpose of extracting the service.

Mistake 2: BFF doing too much. In the first iteration, our web BFF started accumulating business logic. "It's just a small validation," someone said. Then another. Within a month, the BFF had 40 lines of business rules that should have lived in the domain service. We had to refactor it back out.

Mistake 3: Not investing in observability early. With a monolith, you can follow a request in a single log stream. With distributed services, you need distributed tracing from day one. We added OpenTelemetry in month three. Should have been day one.

Mistake 4: Underestimating data consistency. Going from a single database transaction to eventual consistency across services was a mental shift for the whole team. We had to implement the saga pattern for the order-payment-inventory flow, and it took three iterations to get right.

The Trade-offs Nobody Tells You About

Let me be real about what you're signing up for:

Operational complexity goes up significantly. You now have 6 services instead of 1. That's 6 CI/CD pipelines, 6 sets of logs, 6 health checks, 6 things that can fail independently.
Local development gets harder. Docker Compose became mandatory. Onboarding a new developer went from "clone and run" to a 45-minute setup process (which we eventually automated).
Network is now a failure mode. Service-to-service calls can timeout, fail, or return stale data. You need retry logic, circuit breakers, and fallback strategies everywhere.
Testing changes fundamentally. Unit tests are easy. Integration tests across services? That requires contract testing (we used Pact) and a proper staging environment.

But here's the other side:

Teams move independently. The payments squad deploys three times a day without coordinating with anyone.
Scaling is surgical. Black Friday? Scale the order service and payment service. Leave the admin BFF alone.
Failure is isolated. The notification service went down for 20 minutes last month. Orders kept flowing. In the monolith era, that would have been a full outage.

The Honest Takeaway

If your monolith is working and your team is small, don't migrate. Seriously. A well-structured monolith beats a poorly implemented distributed system every single day. We migrated because we had real, measurable pain — not because microservices were trendy.

But if you're feeling that pain — deploy fear, team bottlenecks, scaling ceilings — then a thoughtful migration using BFF + hexagonal architecture + strangler fig is a battle-tested approach. Just go in with your eyes open about the trade-offs, invest in observability from day one, and for the love of everything good in engineering, keep business logic out of the BFF.

The best architecture is the one that lets your team ship with confidence. Everything else is details.

Why Most Backend Systems Fail at Scale (And the Mistakes I've Seen Firsthand)

Tue, 10 Mar 2026 00:00:00 GMT

I've spent over six years building and scaling backend systems, and I can tell you this with absolute confidence: most systems don't fail because of some exotic edge case. They fail because of well-known mistakes that teams keep repeating. Not because engineers are bad — because these problems are invisible at low traffic and only become lethal at scale.

Here are seven mistakes I've seen kill production systems, what actually happened, and how we fixed them. Every single one of these is from a real project.

1. The N+1 Query That Nobody Noticed

What happened: We had an endpoint that returned a list of orders with their associated products. At 50 orders, it was fine — 200ms response time. At 500 orders during a flash sale, it took 14 seconds and the database CPU hit 98%.

The ORM was doing exactly what we asked: load orders, then for each order, load its products. One query for orders, 500 queries for products. Classic N+1.

What we did: The immediate fix was eager loading — one query with a JOIN. Response time dropped to 180ms regardless of order count.

// Before: N+1 disaster
const orders = await orderRepo.find({ where: { userId } });
// This triggers 1 query per order to load products

// After: single query with relation loading
const orders = await orderRepo.find({
  where: { userId },
  relations: ['items', 'items.product'],
});

But the real fix was adding query logging in staging with a threshold alert. Any endpoint that executes more than 10 queries per request gets flagged automatically. We catch N+1s before they reach production now.

Lesson: ORMs make N+1 queries easy to write and hard to spot. You need automated detection, not just code reviews.

2. Missing Circuit Breakers

What happened: Our order service called a third-party shipping rate API. One afternoon, that API started responding with 30-second timeouts instead of its usual 200ms. Every single request to our checkout flow queued up waiting for shipping rates. Thread pool exhausted. Our entire checkout was down — not because OUR system broke, but because someone else's did.

What we did: We implemented circuit breakers using a pattern similar to Netflix's Hystrix (we used opossum in Node.js):

import CircuitBreaker from 'opossum';

const shippingBreaker = new CircuitBreaker(fetchShippingRates, {
  timeout: 3000,
  errorThresholdPercentage: 50,
  resetTimeout: 30000,
});

shippingBreaker.fallback(() => ({
  rates: getCachedShippingRates(),
  isFallback: true,
}));

When the shipping API fails, the circuit opens, and we serve cached rates with a flag telling the frontend these are estimates. Customers keep checking out. We reconcile actual shipping costs asynchronously.

Lesson: Every external dependency is a liability. If you don't have a fallback strategy for each one, you've coupled your uptime to theirs.

3. Shared Database Between Services

What happened: Two services — orders and inventory — both read and wrote to the same PostgreSQL database. The inventory service ran a heavy analytical query every 5 minutes for stock reports. During peak hours, that query locked rows that the order service needed to update. Orders started failing with lock timeout errors.

What we did: We gave each service its own database. Inventory changes get published as events via SNS, and the order service maintains its own read model of available stock. Yes, it's eventually consistent. But "eventually consistent with working checkout" beats "strongly consistent with failing checkout" every time.

Lesson: A shared database is a shared fate. If two services share a database, they're not two services — they're a distributed monolith.

4. No Backpressure on Queues

What happened: We used SQS for processing order confirmations. Each message triggered an email, a PDF invoice generation, and an inventory update. During a promotional event, 50,000 orders came in within an hour. The SQS queue backed up to 200,000 messages (some orders generated multiple messages). Our consumer tried to process all of them at full speed, overwhelmed the email service, and started throwing 429 rate limit errors. Failed messages went to the dead-letter queue. 12,000 customers didn't get their confirmation emails.

What we did: Three changes:

Concurrency limits on consumers: Instead of processing as fast as possible, we capped at 20 concurrent messages per consumer instance.
Rate limiting per downstream dependency: We added a token bucket rate limiter before calling the email API.
Exponential backoff with jitter: Failed messages get retried with increasing delays plus randomization to avoid thundering herd on recovery.

// SQS consumer with controlled concurrency
const consumer = Consumer.create({
  queueUrl: process.env.ORDER_QUEUE_URL,
  batchSize: 10,
  visibilityTimeout: 60,
  handleMessage: async (message) => {
    await rateLimiter.acquire('email-service', { maxPerSecond: 50 });
    await processOrderConfirmation(JSON.parse(message.Body));
  },
});

Lesson: A queue without backpressure is just a way to convert a traffic spike into a cascading failure with extra steps.

5. Ignoring Cold Starts

What happened: We moved a critical service to AWS Lambda for cost savings. Average response time: 80ms. But after periods of inactivity, the first request took 4-6 seconds due to cold starts. This was the authentication service. Users experienced random 5-second login delays. Support tickets piled up.

What we did: We evaluated three options:

Provisioned concurrency: Keeps Lambda instances warm. Effective but expensive — defeated the cost savings.
Warm-up pings: A CloudWatch scheduled event hitting the function every 5 minutes. Cheap but unreliable under concurrent load — only keeps one instance warm.
Move back to ECS Fargate: What we actually did. The auth service needs consistent sub-100ms latency. Lambda's model doesn't fit that requirement.

Lesson: Serverless is fantastic — for the right workloads. Latency-sensitive, always-on services aren't it. Match the compute model to the actual requirements, not the trend.

6. Cache Stampede

What happened: We cached our product catalog in Redis with a 5-minute TTL. At the exact moment the cache expired, 3,000 concurrent requests hit the endpoint. All 3,000 saw a cache miss. All 3,000 queried the database simultaneously. Database connection pool exhausted. 500 errors for everyone.

What we did: We implemented a cache lock pattern (sometimes called "probabilistic early expiration"):

async function getProductCatalog(categoryId: string): Promise<Product[]> {
  const cacheKey = `catalog:${categoryId}`;
  const cached = await redis.get(cacheKey);

  if (cached) {
    const { data, expiry, delta } = JSON.parse(cached);
    const now = Date.now();

    // Probabilistic early recomputation
    // As we approach expiry, increasingly likely to refresh
    if (now - delta * Math.log(Math.random()) < expiry) {
      return data;
    }
  }

  // Distributed lock: only one process rebuilds
  const lockKey = `lock:${cacheKey}`;
  const acquired = await redis.set(lockKey, '1', 'EX', 30, 'NX');

  if (!acquired) {
    // Another process is rebuilding; serve stale if available
    if (cached) return JSON.parse(cached).data;
    // No stale data; wait briefly and retry
    await sleep(100);
    return getProductCatalog(categoryId);
  }

  try {
    const products = await productRepo.findByCategory(categoryId);
    const ttl = 300; // 5 minutes
    await redis.set(cacheKey, JSON.stringify({
      data: products,
      expiry: Date.now() + ttl * 1000,
      delta: ttl * 1000 * 0.1,
    }), 'EX', ttl + 60); // Extra buffer for stale serving
    return products;
  } finally {
    await redis.del(lockKey);
  }
}

Only one process rebuilds the cache. Everyone else gets slightly stale data. In an e-commerce catalog, nobody notices 30-second-old product data. Everyone notices a 500 error.

Lesson: Cache expiration is a coordinated event that can create thundering herds. Design for it explicitly.

7. Not Designing for Failure

What happened: This is the meta-mistake that encompasses all the others. We built our first version of the platform assuming everything would work. Happy path everywhere. No retries, no fallbacks, no graceful degradation. The system worked perfectly in development and staging. Production taught us otherwise within the first week.

What we did: We adopted a principle: every external call must answer three questions before it's written:

What happens if this call fails?
What happens if this call is slow (10x normal latency)?
What happens if this call returns unexpected data?

We codified this into our code review checklist. Pull requests that add external calls without answering these three questions get sent back.

We also started running game days — intentionally injecting failures in staging using AWS Fault Injection Simulator. You'd be amazed at how many assumptions break when you kill a database replica or add 500ms latency to an internal service.

Lesson: Failure isn't an edge case. It's a feature of distributed systems. If you haven't designed for it, you haven't designed the system.

The Pattern Behind All These Failures

Look at all seven mistakes. There's a common thread: they're all invisible at low scale. N+1 queries are fast when N is small. Missing circuit breakers don't matter when dependencies are healthy. Cache stampedes don't happen with 10 concurrent users.

This is why load testing and chaos engineering aren't optional luxuries — they're how you find these problems before your customers do. We run load tests that simulate 3x our peak traffic before every major release. We run chaos experiments monthly. It's not paranoia. It's professionalism.

The systems that survive at scale aren't the ones built by the smartest engineers. They're the ones built by engineers who assumed everything would break and planned accordingly. Build for the failure case first. The happy path takes care of itself.

Redis in Production: Caching Strategies That Actually Work (And Some That Don't)

Thu, 05 Mar 2026 00:00:00 GMT

I've been running Redis in production on e-commerce platforms for several years now, and I can tell you that most caching tutorials leave out the parts that actually matter. They show you SET key value EX 300 and call it a day. Real production caching is about trade-offs, invalidation strategies, failure modes, and knowing when NOT to cache something.

This is everything I wish someone had told me before I learned it the hard way.

The Three Core Strategies (And When Each One Fits)

Cache-Aside (Lazy Loading)

This is the most common pattern, and for good reason. The application checks the cache first. On a miss, it reads from the database, writes to the cache, and returns the data.

async function getProduct(id: string): Promise<Product> {
  const cacheKey = `product:${id}`;

  // 1. Check cache
  const cached = await redis.get(cacheKey);
  if (cached) return JSON.parse(cached);

  // 2. Cache miss — read from DB
  const product = await productRepo.findById(id);
  if (!product) throw new NotFoundException();

  // 3. Populate cache
  await redis.set(cacheKey, JSON.stringify(product), 'EX', 600);

  return product;
}

When it works: Read-heavy workloads where stale data is acceptable for short periods. Product catalogs, user profiles, configuration data.

When it doesn't: Data that changes frequently and must be immediately consistent. If you cache a user's cart with cache-aside and they add an item, they'll see the old cart until the TTL expires (unless you actively invalidate — more on that later).

Trade-off: Simple to implement, but every cache miss is a slow request. Cold cache after a deploy or Redis restart means a thundering herd to the database.

Write-Through

Every write goes to both the cache and the database simultaneously. Reads always hit the cache.

async function updateProduct(id: string, data: UpdateProductDto): Promise<Product> {
  // 1. Update database
  const product = await productRepo.update(id, data);

  // 2. Update cache (synchronously — part of the write path)
  const cacheKey = `product:${id}`;
  await redis.set(cacheKey, JSON.stringify(product), 'EX', 600);

  return product;
}

When it works: Data that's read frequently and updated occasionally, AND where you need strong read consistency. Session data, feature flags, pricing rules.

When it doesn't: Write-heavy workloads. You're paying the latency cost of a Redis write on every single database write, even for data nobody might read.

Trade-off: Cache is always fresh, but writes are slower. And you still need a TTL as a safety net — if the write-through fails silently, the cache diverges forever without one.

Write-Behind (Write-Back)

Writes go to the cache first, and the cache asynchronously flushes to the database. This is the fastest for writes but the most dangerous.

When it works: High-frequency counters, analytics events, real-time metrics — where losing a few data points is acceptable.

When it doesn't: Anything transactional. If Redis crashes before flushing to the database, that data is gone. I would never use this for orders, payments, or anything financial.

Trade-off: Maximum write performance, but you're accepting potential data loss. In my experience, this pattern is rarely the right choice for core business data. We use it for view counts and analytics pipelines — that's about it.

TTL Strategies Beyond "Set It to 5 Minutes"

The number one TTL anti-pattern I see is slapping the same TTL on everything. Your product catalog and your real-time inventory count have wildly different freshness requirements. Treat them differently.

Here's how we think about TTL design:

Static reference data (countries, categories, tax rates): TTL of 24 hours. Invalidate explicitly on change.
Semi-static data (product details, user profiles): TTL of 10-30 minutes. Stale data is annoying but not harmful.
Volatile data (inventory counts, pricing): TTL of 30-60 seconds. Or don't cache at all — sometimes the database is the right answer.
Session/auth data: TTL matches the session duration. No more, no less.

We also use adaptive TTLs based on access patterns:

async function getWithAdaptiveTTL(key: string, fetcher: () => Promise<any>): Promise<any> {
  const cached = await redis.get(key);
  if (cached) {
    // Extend TTL on access (most-recently-used stays warm)
    const currentTTL = await redis.ttl(key);
    if (currentTTL < 120) {
      await redis.expire(key, 600); // Reset to 10 min on access
    }
    return JSON.parse(cached);
  }

  const data = await fetcher();
  await redis.set(key, JSON.stringify(data), 'EX', 600);
  return data;
}

Frequently accessed data stays warm. Rarely accessed data naturally evicts. Your cache memory stays focused on what matters.

Cache Invalidation: The Real Hard Problem

Phil Karlton said there are two hard things in computer science: cache invalidation and naming things. He wasn't wrong about the first one.

Here are the invalidation patterns we use, from simplest to most robust:

TTL-Based Expiration (Passive)

Let it expire naturally. This is fine when stale data is acceptable. It's simple, and simple is underrated.

Explicit Invalidation (Active)

When data changes, delete the cache key:

async function updateProduct(id: string, data: UpdateProductDto): Promise<Product> {
  const product = await productRepo.update(id, data);

  // Delete, don't update — let the next read repopulate
  await redis.del(`product:${id}`);

  // Don't forget related caches!
  await redis.del(`catalog:${product.categoryId}`);
  await redis.del(`search:products:*`); // Pattern deletion for search caches

  return product;
}

Important: Delete, don't update. If you update the cache and the database write fails or rolls back, your cache has phantom data. Delete is idempotent and safe.

But notice the search:products:* — that's where invalidation gets ugly. A product change might affect category listings, search results, recommendation feeds, and homepage features. Missing even one creates a stale data bug that's incredibly hard to track down.

Event-Driven Invalidation

This is what we settled on for anything non-trivial. Database changes publish events, and a dedicated cache invalidation consumer handles the cleanup:

// Event handler — separate service/consumer
@OnEvent('product.updated')
async handleProductUpdate(event: ProductUpdatedEvent) {
  const { productId, categoryId, previousCategoryId } = event;

  const keysToInvalidate = [
    `product:${productId}`,
    `catalog:${categoryId}`,
    `homepage:featured`,
  ];

  // If category changed, invalidate old category too
  if (previousCategoryId && previousCategoryId !== categoryId) {
    keysToInvalidate.push(`catalog:${previousCategoryId}`);
  }

  await Promise.all(keysToInvalidate.map(key => redis.del(key)));
}

This centralizes invalidation logic. When a new cache is added, you add its invalidation rule to the event handler — one place to maintain instead of scattered redis.del() calls across the codebase.

Redis Cluster on AWS ElastiCache: What We Learned

Running Redis on ElastiCache in cluster mode has some sharp edges:

Key distribution matters. Redis Cluster uses hash slots. If you use MGET across keys that land on different slots, it fails. Use hash tags ({product}:123, {product}:456) to colocate related keys on the same shard.
Failover isn't instant. When a primary node fails, ElastiCache promotes a replica. This takes 10-30 seconds. Your application needs to handle connection errors gracefully during failover. We use ioredis with retryStrategy and reconnectOnError configured properly.
Memory management is critical. Set maxmemory-policy to allkeys-lru for caching workloads. We've seen teams leave it at noeviction (the default) and then wonder why Redis starts returning errors when it fills up.
Monitor replication lag. If your read replicas fall behind, reads from replicas serve stale data. We alert at 1 second of replication lag.

When NOT to Cache

This is the section most articles skip. Here's when caching makes things worse:

Write-heavy, read-light data. If you write 10x more than you read, you're paying the cache write cost without the read benefit.
Highly personalized data with low reuse. Caching a unique recommendation feed per user sounds good until you realize your cache hit rate is 3%. You're using Redis as an expensive, volatile second database.
Data that must be real-time consistent. Account balances, inventory for limited-edition drops, auction bids. The cost of stale data exceeds the cost of a database query.
Large objects that rarely repeat. Caching a 2MB report PDF in Redis is a waste of memory. Put it in S3.

The rule of thumb: If your cache hit rate is below 80%, investigate whether that cache is earning its keep. Below 50%, it's probably hurting more than helping.

Monitoring Cache Health

A cache you don't monitor is a cache waiting to betray you. Here's what we track:

Hit rate (target: >90%): cache_hits / (cache_hits + cache_misses). We expose this as a Prometheus metric and alert if it drops below 80%.
Latency percentiles: p50, p95, p99 for Redis operations. If p99 spikes, you might have a hot key or a network issue.
Memory usage: Percentage of available memory used. Alert at 80%.
Eviction rate: If keys are being evicted faster than expected, your cache is undersized for the workload.
Connection count: Suddenly high connection counts can indicate a connection leak in your application.

We use OpenTelemetry to instrument all Redis calls and ship traces to our observability stack. Every cache operation gets a span with the key pattern (not the full key — you don't want high-cardinality labels), the operation type, and whether it was a hit or miss.

// Simplified OTEL instrumentation wrapper
async function cachedGet<T>(key: string, fetcher: () => Promise<T>, ttl: number): Promise<T> {
  const span = tracer.startSpan('cache.get', {
    attributes: { 'cache.key_pattern': key.split(':')[0], 'cache.ttl': ttl },
  });

  try {
    const cached = await redis.get(key);
    if (cached) {
      span.setAttribute('cache.hit', true);
      cacheHitCounter.inc({ pattern: key.split(':')[0] });
      return JSON.parse(cached);
    }

    span.setAttribute('cache.hit', false);
    cacheMissCounter.inc({ pattern: key.split(':')[0] });
    const data = await fetcher();
    await redis.set(key, JSON.stringify(data), 'EX', ttl);
    return data;
  } finally {
    span.end();
  }
}

The Bottom Line

Redis is one of the most powerful tools in a backend engineer's toolkit, but it's not magic. It's a trade-off machine: memory for speed, consistency for performance, simplicity for resilience. Every caching decision should start with "what's the cost of stale data?" and "what's the cost of a cache miss?" — not with "let's cache everything and hope for the best."

Start with cache-aside for most things. Add write-through only where consistency matters. Use event-driven invalidation for anything complex. Monitor your hit rates religiously. And never, ever forget: the fastest cache miss is the one you prevent by not caching data that shouldn't be cached in the first place.

Designing Systems When You Don't Control All the Dependencies

Sat, 28 Feb 2026 00:00:00 GMT

There's a moment in every engineer's career where you realize something uncomfortable: the system you're responsible for is only as reliable as its weakest external dependency. And you don't control that dependency.

Maybe it's a payment gateway that times out during peak hours. Maybe it's a legacy inventory system maintained by a team in another timezone that deploys on Fridays. Maybe it's a shipping provider whose API returns 500s every time they push a "minor update."

I've lived through all of these. And here's what I've learned: you don't design resilient systems by hoping everything works. You design them by assuming everything will break.

The Trust Boundary Concept

Before diving into patterns, let's establish a mental model I use constantly: the trust boundary.

Every system has a boundary between what you control and what you don't. Inside the boundary, you own the code, the deployments, the monitoring, and the on-call rotation. Outside the boundary, you're at the mercy of someone else's priorities, release schedule, and definition of "reliable."

The mistake most teams make is treating external dependencies as if they're inside the trust boundary. They call a third-party API and assume it'll respond in 200ms. They integrate with an internal legacy service and assume the contract won't change. They trust.

Don't trust. Verify, isolate, and plan for failure.

Here's how I draw trust boundaries in practice:

Tier 1 (full trust): Services my team owns and deploys. We control SLAs.
Tier 2 (conditional trust): Internal services from other teams. We have some influence but no control.
Tier 3 (zero trust): Third-party APIs, legacy systems, anything behind a VPN we can't monitor.

Each tier gets a different resilience strategy. Tier 3 dependencies get the full treatment.

Circuit Breaker Pattern: Stop Hammering Dead Services

The circuit breaker is the first pattern you should reach for when dealing with unreliable dependencies. The concept is borrowed from electrical engineering: when a downstream service is failing, stop sending requests to it.

Here's a simplified implementation I've used in Node.js services:

class CircuitBreaker {
  private failures = 0;
  private lastFailure: number = 0;
  private state: 'closed' | 'open' | 'half-open' = 'closed';

  constructor(
    private readonly threshold: number = 5,
    private readonly resetTimeout: number = 30_000
  ) {}

  async execute<T>(fn: () => Promise<T>, fallback: () => T): Promise<T> {
    if (this.state === 'open') {
      if (Date.now() - this.lastFailure > this.resetTimeout) {
        this.state = 'half-open';
      } else {
        return fallback();
      }
    }

    try {
      const result = await fn();
      this.onSuccess();
      return result;
    } catch (error) {
      this.onFailure();
      return fallback();
    }
  }

  private onSuccess(): void {
    this.failures = 0;
    this.state = 'closed';
  }

  private onFailure(): void {
    this.failures++;
    this.lastFailure = Date.now();
    if (this.failures >= this.threshold) {
      this.state = 'open';
    }
  }
}

In an e-commerce system I worked on, we wrapped every payment gateway call with a circuit breaker. When Stripe had intermittent issues (yes, even Stripe has bad days), the circuit would open after 5 consecutive failures. Instead of queuing thousands of failed payment attempts and making users stare at a spinner, we'd immediately route to the fallback flow: show a "payment processing delayed" message and queue the charge for retry.

Key insight: The circuit breaker doesn't fix the problem. It prevents cascade failures and gives users a better experience during outages.

Bulkhead Isolation: Don't Let One Failure Sink the Ship

Bulkheads in ships are watertight compartments. If one compartment floods, the ship doesn't sink. The same principle applies to software.

In a real scenario: we had an e-commerce platform where the shipping rate calculator, the inventory checker, and the payment processor all shared the same HTTP connection pool. When the shipping provider started responding slowly, it consumed all available connections. Suddenly, payments couldn't process either. One bad dependency took down an unrelated critical path.

The fix was bulkhead isolation:

Separate connection pools per external dependency
Separate thread pools (or in Node.js, separate queue workers) for different categories of work
Separate timeouts tuned to each dependency's expected behavior

const paymentClient = new HttpClient({
  maxConnections: 20,
  timeout: 5_000,       // payments should be fast
  retries: 2,
});

const shippingClient = new HttpClient({
  maxConnections: 10,
  timeout: 15_000,      // shipping APIs are notoriously slow
  retries: 3,
});

const inventoryClient = new HttpClient({
  maxConnections: 15,
  timeout: 3_000,       // internal service, should be quick
  retries: 1,
});

This is not over-engineering. This is the difference between "the shipping API is slow today" and "the entire checkout is down."

Timeout Strategies: The Silent Killer

Most production incidents I've debugged didn't start with a crash. They started with a timeout that was either too generous or nonexistent.

Here's my timeout hierarchy for external calls:

Connect timeout: How long to wait for a TCP connection. Keep this aggressive (1-3 seconds). If a service isn't accepting connections, waiting longer won't help.
Read timeout: How long to wait for a response after the connection is established. This depends on the operation, but I rarely go above 10 seconds for synchronous calls.
Total timeout: The overall budget for the entire operation, including retries. This is the one most people forget.

The rule I follow: every HTTP call to an external service MUST have an explicit timeout. No exceptions. Default timeouts in HTTP libraries are usually 30-120 seconds, which is an eternity in a request pipeline.

A practical example: in a checkout flow, the total budget is maybe 8 seconds before the user gets impatient. If you're calling a payment API, an address validator, and a tax calculator in sequence, you don't have 30 seconds per call. You have maybe 2-3 seconds each, with the payment getting the most generous allocation.

Fallback Mechanisms: Graceful Degradation in Practice

Here's where it gets interesting. Fallbacks aren't just about returning cached data. They're about deciding what your system can still do without a dependency.

Real examples from systems I've built:

Payment gateway down: Queue the order, show "processing" status, charge asynchronously when the gateway recovers. The user still completes checkout.
Shipping rate calculator down: Show estimated flat rates based on historical data. Not perfect, but the user can still buy.
Inventory service down: Allow the purchase but flag it for manual review. Accept the risk of overselling a few items vs. blocking all sales.
Recommendation engine down: Show popular products instead of personalized ones. Nobody notices.

The pattern:

async function getShippingRates(order: Order): Promise<ShippingRate[]> {
  try {
    return await shippingCircuitBreaker.execute(
      () => shippingClient.calculateRates(order),
      () => getCachedFlatRates(order.destination)
    );
  } catch {
    return getDefaultFlatRates();  // last resort
  }
}

The business conversation matters here. Fallbacks have trade-offs. Showing flat rates when the real rates are different means you might lose money on some shipments. Queue-and-charge means you might charge a card that gets declined later. These are business decisions, not just technical ones. Have this conversation with product before the outage, not during.

Contract Testing: Catch Breaks Before Production

One of the most effective strategies I've adopted is consumer-driven contract testing. Instead of praying that the shipping API doesn't change their response format, we codify what we expect and test it regularly.

Tools like Pact let you define contracts:

describe('Shipping API Contract', () => {
  it('should return rates with expected structure', async () => {
    const response = await provider.executeTest(async (mockServer) => {
      const client = new ShippingClient(mockServer.url);
      const rates = await client.getRates(testOrder);

      expect(rates).toEqual(
        expect.arrayContaining([
          expect.objectContaining({
            carrier: expect.any(String),
            price: expect.any(Number),
            estimatedDays: expect.any(Number),
          }),
        ])
      );
    });
  });
});

This won't prevent the external API from breaking. But it'll tell you immediately when the contract is violated, instead of discovering it through production errors at 2 AM.

The Trade-Off: Resilience vs. Simplicity

Here's the thing I want to be honest about: all of this adds complexity. Circuit breakers, bulkheads, fallbacks, contract tests — they all add code, configuration, and cognitive overhead.

Not every system needs all of these patterns. A side project calling one external API? Just add a timeout and handle errors gracefully. An e-commerce platform processing thousands of orders per hour across a dozen external services? You need the full toolkit.

My rule of thumb:

If downtime of the dependency costs money: circuit breaker + fallback + monitoring
If downtime causes data loss: bulkhead isolation + retries with idempotency
If the dependency is owned by another team: contract tests + explicit SLAs
If it's a third-party you can't influence at all: all of the above

Closing Thoughts

The hardest lesson in system design isn't learning patterns. It's accepting that you can't control everything. Your payment provider will have outages. The legacy system will change without warning. The team across the building will deploy a breaking change on a Friday.

Your job isn't to prevent these failures. It's to design systems that handle them with grace. Every resilience pattern is a bet: "When this dependency fails — and it will — here's how we'll keep going."

The engineers who build truly reliable systems aren't the ones who write the most clever code. They're the ones who've been burned enough times to plan for the fire.

The Role of a Tech Lead Beyond Writing Code

Fri, 20 Feb 2026 00:00:00 GMT

When I first became a Tech Lead, I thought the job was simple: be the best coder on the team and make the big technical decisions. I was wrong about pretty much everything.

The role of a Tech Lead is one of the most misunderstood positions in software engineering. It's not a promotion for writing great code. It's a fundamentally different job. And nobody prepares you for it.

After years of leading teams across multiple production systems, here's what the role actually looks like — the parts nobody talks about in job descriptions.

Architectural Decisions Are Organizational Decisions

Every architectural choice you make has organizational consequences. Choose microservices? Now you need team boundaries, API contracts, and deployment coordination. Choose a monolith? Now you need modular discipline and merge conflict resolution strategies.

I've seen teams adopt event-driven architectures because it was "the right pattern" without considering that nobody on the team had experience with eventual consistency. The architecture was sound on paper. In practice, it created months of debugging sessions and a backlog of data inconsistency bugs.

As a Tech Lead, your job isn't to pick the theoretically best architecture. It's to pick the best architecture for your team, your timeline, and your business constraints.

This means asking questions like:

Can our team maintain this in 6 months when I'm not reviewing every PR?
Does this architecture match our deployment capabilities?
If we hire two junior devs next quarter, will they be able to contribute to this?
What's the cost of being wrong about this choice?

The best architectural decision I ever made wasn't choosing an elegant pattern. It was choosing a boring, well-understood one that the entire team could work with independently. Boring is underrated.

Mentoring: Your Multiplier Effect

Here's a math problem: you can write 8 hours of code per day, or you can spend 2 hours mentoring three engineers who then each write 6 effective hours of code per day. The first option produces 8 hours of output. The second produces 18.

This is the multiplier effect, and it's the single most impactful thing a Tech Lead does.

But mentoring isn't just answering questions or doing pair programming (though both matter). Real mentoring means:

Code review as teaching: Don't just approve or request changes. Explain why something should be different. Link to documentation. Show the consequences of the current approach. A PR comment that says "use dependency injection here" is worthless. A comment that says "if we inject this dependency, we can test the payment flow without hitting Stripe — here's how" teaches a principle.
Delegating the hard problems: This is counterintuitive. Your instinct is to take the complex tasks because you'll do them faster. Resist that instinct. Give the challenging work to growing engineers with your support. Yes, it'll take longer. Yes, the first implementation won't be as clean. But now you have two people who can solve that kind of problem instead of one.
Creating safe failure spaces: Engineers who are afraid to make mistakes don't grow. I explicitly tell my team: "This is a safe area to experiment. If it breaks, we'll fix it together." The best engineers I've mentored made spectacular mistakes early because they felt safe to try.

One of my proudest moments as a Tech Lead wasn't shipping a feature. It was watching a mid-level engineer I'd been mentoring for a year lead an architectural discussion and make better decisions than I would have.

Navigating Product vs. Engineering Tensions

Every sprint planning is a negotiation between what the product team wants and what the engineering team knows is sustainable. As Tech Lead, you're the translator between these two worlds.

Product says: "We need this feature by next sprint." Engineering knows: the data model doesn't support it, and bolting it on will create debt that slows everything down for months.

Your job isn't to say "no." Your job is to say "here are the options with their trade-offs."

A framework I use constantly:

Option A: Do it the fast way. Ship in 1 sprint. Creates X amount of tech debt. Estimated cost to fix later: Y sprints. Risk: Z.
Option B: Do it properly. Ship in 2 sprints. No debt. Enables future features P and Q without rework.
Option C: Ship a reduced scope in 1 sprint. Addresses 80% of user need. Proper architecture. Iterate on remaining 20% in the next sprint.

Nine times out of ten, option C wins. But you can only present option C if you deeply understand both the business goal and the technical landscape.

The Tech Leads who get burned out are the ones who just say "yes" to everything. Learn to present options. Let the stakeholders decide with full information. That's not resistance — that's partnership.

Saying "No" to Scope Creep

Scope creep is the silent project killer. It doesn't arrive as one big change. It arrives as a series of "small" additions that seem reasonable in isolation.

"Can we also add filtering to this page?" "What if we supported CSV export too?" "Could this API also handle the admin use case?"

Each request is 2-3 days of work. Ten of them is a month of unplanned work that nobody budgeted for.

As a Tech Lead, you're often the only person in the room who can see the cumulative technical cost. It's your responsibility to make it visible.

I keep a running document I call the "scope ledger." Every addition gets logged with its estimated effort. When stakeholders ask why the project is delayed, I can point to 15 "small" additions that added up to 6 weeks of extra work. Data changes conversations.

Technical Debt as a Business Conversation

Here's something that took me years to learn: engineers talk about technical debt in engineering terms. "The codebase is messy." "We need to refactor this module." "Our test coverage is too low." These statements mean nothing to business stakeholders.

Technical debt needs to be translated into business impact:

Don't say: "We need to refactor the payment module."
Say: "The payment module takes 3 days to add any new payment method. Our competitors are adding new methods quarterly. Refactoring reduces that to half a day."
Don't say: "Our test coverage is 20%."
Say: "We've had 4 production incidents this quarter that would have been caught by tests. Each incident cost us an average of $X in lost transactions and Y hours of engineering time."
Don't say: "The codebase is tightly coupled."
Say: "Adding the feature you want requires modifying 12 files across 4 services because of how things are connected. This is why estimates keep being higher than expected."

When you frame debt as a business problem with measurable costs, you get budget to fix it. When you frame it as an engineering preference, you get told to "make it work anyway."

The Loneliness of Leading

Nobody warns you about this part. As a Tech Lead, you occupy a strange middle ground. You're not management, but you're not "one of the developers" anymore either. You're in meetings your team isn't in. You know about reorgs, budget cuts, and strategic pivots before anyone else. You can't always share what you know.

Your team will sometimes be frustrated with decisions you've made or agreed to. They won't always have the context for why. And you can't always give them that context.

There are days where you're the shield between your team and organizational chaos, and nobody will ever know. The best compliment a Tech Lead can receive isn't "great architecture." It's a team member saying "I don't know why, but things just work smoothly here." That smoothness is you absorbing friction they never see.

Find a peer group. Talk to other Tech Leads. This role is isolating if you don't have people who understand the specific pressures you're under.

When to Code and When to Step Back

This is the hardest calibration. Code too much, and you become a bottleneck — the team depends on you for critical path work. Code too little, and you lose touch with the codebase, making your architectural guidance increasingly theoretical.

My current balance:

I write code for: proof of concepts for architectural decisions, critical infrastructure pieces where the patterns need to be established, debugging production issues that require deep system knowledge.
I don't write code for: feature work that a team member could do (even if slower), anything on the critical path of a sprint commitment, tasks that would be better as learning opportunities for someone else.

A practical rule: if removing me from the sprint board would block delivery, I'm coding too much of the wrong things. The team should be able to ship without me writing a single line of feature code.

The Transition: Best Coder to Best Enabler

The most difficult shift in becoming a Tech Lead is redefining what "productive" means. For your entire career, productivity meant lines of code, features shipped, problems solved with your own hands.

Now, productivity means:

A junior engineer who ships their first production feature with confidence
An architecture that enables the team to move independently
A code review culture where engineers teach each other
A sprint that finishes on time because you identified risks early
A production system that handles failure gracefully because you invested in resilience patterns

Your code output will drop. Your impact will multiply. That's not a trade-off. That's the entire point.

Closing Thoughts

Being a Tech Lead changed how I think about software. I used to optimize for elegant code. Now I optimize for team velocity, sustainable systems, and people growth.

If you're transitioning into this role, here's my honest advice: let go of being the best coder in the room. Invest in being the person who makes everyone else better. The code you write will be forgotten. The engineers you grow will build things you never could have built alone.

That's not a demotion. That's leverage.

AI in the Developer Workflow: What Actually Works (And What's Just Hype)

Sun, 15 Feb 2026 00:00:00 GMT

Let me be direct: AI is not going to replace developers. But it's also not a toy. The truth, as usual, is somewhere in the middle — and most of the discourse around AI in development is either breathless hype or dismissive cynicism. Both are wrong.

I've been using AI tools daily in my development workflow for over a year now. Not as an experiment. In production work, leading teams, shipping real systems. Here's my honest, no-hype assessment of what actually works, what doesn't, and what you should be thinking about.

What AI Is Genuinely Good At

Let's start with the wins. These are areas where AI tools have measurably improved my workflow:

Boilerplate and Repetitive Code

This is where AI shines brightest. Writing DTOs, mapping functions, CRUD endpoints, configuration files, type definitions from API specs — the kind of code where the pattern is well-established and you're essentially translating a known structure into syntax.

// "Generate a DTO for the order response with these fields"
// AI produces this in seconds vs. minutes of typing:
interface OrderResponseDto {
  id: string;
  status: OrderStatus;
  items: OrderItemDto[];
  total: number;
  currency: string;
  createdAt: string;
  shippingAddress: AddressDto;
  estimatedDelivery: string | null;
}

Is this revolutionary? No. Does it save 20 minutes of tedious typing multiple times per day? Absolutely. That adds up.

Test Scaffolding

This is probably my single favorite use case. AI is remarkably good at generating test scaffolds — the describe blocks, the setup, the basic happy-path and error cases. It gets the structure right roughly 80% of the time.

// I describe what the function does, AI generates the scaffold:
describe('OrderService.calculateTotal', () => {
  let service: OrderService;
  let mockTaxCalculator: jest.Mocked<TaxCalculator>;

  beforeEach(() => {
    mockTaxCalculator = { calculate: jest.fn() };
    service = new OrderService(mockTaxCalculator);
  });

  it('should sum item prices correctly', () => { /* ... */ });
  it('should apply tax from calculator', () => { /* ... */ });
  it('should handle empty order', () => { /* ... */ });
  it('should throw on negative quantities', () => { /* ... */ });
  it('should round to 2 decimal places', () => { /* ... */ });
});

I still write the actual assertions and edge cases myself. But having the scaffold, the mocks set up, and the test organization ready saves significant time. It's like having someone set up the workbench before you start building.

Exploring Unfamiliar Codebases

When I join a new project or need to understand a library I've never used, AI is an excellent exploration partner. "Explain how this middleware chain works." "What does this Terraform module provision?" "Walk me through the authentication flow in this codebase."

It's not always right. But it's faster than reading documentation from scratch, and it gives me a starting point for my own investigation. I treat it like asking a colleague who's worked on the codebase before — useful input, but I verify anything critical.

Documentation and Commit Messages

Writing JSDoc comments, README sections, API documentation, and even thoughtful commit messages. AI handles this surprisingly well because documentation is fundamentally about describing what already exists — and the code is right there as context.

Refactoring Suggestions

"Here's a 200-line function. Suggest how to break it down." The suggestions aren't always production-ready, but they're a useful starting point for thinking about decomposition. AI can spot extract-method opportunities faster than I can read through a long function.

What AI Is Terrible At

Now the important part. These are areas where trusting AI output will hurt you:

Architecture Decisions

AI cannot make architectural decisions for your system. It doesn't understand your team's capabilities, your deployment constraints, your business timeline, your compliance requirements, or the political dynamics that influence technical choices.

I've seen engineers ask AI "should I use microservices or a monolith?" and treat the response as gospel. The response will be coherent. It'll mention trade-offs. It'll sound authoritative. And it'll be generic advice that ignores everything that actually matters about your specific situation.

Architecture requires context that doesn't fit in a prompt. It requires understanding organizational dynamics, team skills, operational maturity, and business trajectory. AI has none of this.

Business Logic

The core logic that makes your system valuable — the pricing algorithms, the matching rules, the compliance checks, the domain-specific calculations — this is exactly where AI is most dangerous.

Why? Because it'll generate code that looks correct. It compiles. It passes basic tests. But it encodes assumptions about your business domain that may be subtly wrong. And subtle bugs in business logic are the kind that cost real money and erode user trust.

I had a case where AI-generated code for a discount calculation handled percentage-based discounts correctly but applied them in the wrong order when stacked with flat-rate discounts. The code looked clean. The tests that AI also generated passed. The bug made it to staging and would have cost thousands in incorrectly priced orders.

Your business logic is your competitive advantage. Write it yourself. Understand every line.

Security

Do not let AI write your authentication flows, authorization logic, input validation, or anything related to security. AI models are trained on a massive corpus of code, and a significant portion of that code has security vulnerabilities.

AI will generate SQL queries without parameterization. It'll create JWT validation that doesn't check expiration. It'll write file upload handlers without size limits or type checking. Not every time — but often enough that you cannot trust it.

Security is a domain where "works most of the time" is completely unacceptable.

The "AI-Assisted" vs. "AI-Generated" Distinction

This is the framework I use to think about AI in my workflow, and I think it's the most important distinction in this entire conversation.

AI-Assisted: I'm driving. I understand the problem, I know what the solution should look like, and I'm using AI to accelerate the mechanical parts of implementation. I review every line, I make the decisions, AI handles the typing.

AI-Generated: AI is driving. I describe what I want, AI produces the solution, and I ship it with minimal review. I'm essentially outsourcing understanding.

The first approach makes you faster. The second makes you dangerous.

Here's my rule: if you can't explain every line of the AI-generated code, you shouldn't ship it. Not "I roughly understand what it does" — you should be able to explain why each decision was made, what the edge cases are, and how it fails.

If you can't do that, you don't have a solution. You have a liability.

How I Actually Use AI Day-to-Day

Here's my realistic workflow:

Morning code review acceleration: I use AI to get a quick summary of large PRs before doing my detailed review. "Summarize the changes in this PR and flag potential issues." This gives me a map before I dive into the territory. I still review every file, but I know where to focus attention.

Test generation: When writing a new service, I write the implementation first, then ask AI to generate test scaffolds. I modify the tests heavily — adding edge cases, fixing assertions, ensuring the tests actually validate behavior rather than implementation. But the scaffold saves 30-40% of the time.

Exploring new libraries: Before reading full documentation, I ask AI to explain the core concepts and common patterns. This gives me a mental model that makes the docs more digestible. Always followed by reading the actual docs.

Boilerplate acceleration: DTOs, mappers, configuration, migration files. Anything where the pattern is established and I'm translating a known structure into code.

What I never use AI for: Architecture decisions, database schema design, security logic, performance-critical code paths, business rule implementation, or any code where being subtly wrong has significant consequences.

Why Fundamentals Matter MORE With AI, Not Less

Here's the take that might be controversial: AI makes software engineering fundamentals more important, not less.

If you don't understand design patterns, you can't evaluate whether AI's suggested pattern is appropriate. If you don't understand database indexing, you can't spot that AI's generated query will do a full table scan on a million-row table. If you don't understand HTTP semantics, you can't catch that AI used POST where it should have used PUT.

AI amplifies what you already know. If you understand system design deeply, AI makes you dramatically faster. If you don't understand the fundamentals, AI makes you dramatically more efficient at producing broken software.

The developers who are thriving with AI tools are the ones who already had strong fundamentals. They use AI as a power tool, not a crutch. They know enough to:

Evaluate AI output critically
Catch subtle errors that look correct at first glance
Direct AI toward the right solution by asking better questions
Know when NOT to use it — which might be the most important skill of all

The Danger of The Comfort Zone

I see a pattern that concerns me: developers who stop learning because AI can fill the gaps. They don't learn how promises work because AI writes their async code. They don't understand SQL joins because AI generates their queries. They don't learn design patterns because AI structures their code.

This works until it doesn't. And when it doesn't — when you're debugging a race condition at 2 AM, or designing a system that needs to handle 10x traffic, or figuring out why the AI-generated code is causing a memory leak — there's no shortcut. You either understand the fundamentals or you don't.

AI is a tool. You are the engineer. The tool doesn't replace the skill — it requires it.

Closing Thoughts

AI in the developer workflow is genuinely useful. I'm more productive with it than without it. That's not hype — that's my honest experience after extensive daily use.

But the useful parts are the mundane parts. The boilerplate. The scaffolding. The exploration. The documentation. The stuff that was never the hard part of software engineering in the first place.

The hard parts — understanding problems deeply, making architectural trade-offs, writing correct business logic, building secure systems, mentoring teams — those are still entirely human skills. And I don't see that changing anytime soon.

Use AI tools. Get faster at the mechanical parts of development. But invest even more time in understanding the fundamentals. Because the developers who will thrive in an AI-augmented world aren't the ones who can prompt the best. They're the ones who can think the best.

The AI is the power tool. You're still the craftsman. Don't forget which one matters more.

Building Side Projects While Working Full-Time: A System, Not Motivation

Tue, 10 Feb 2026 00:00:00 GMT

Let me be honest with you: motivation is a terrible strategy for shipping side projects. I've been building things on the side for years while holding a demanding full-time tech lead role, and the projects that actually shipped had nothing to do with bursts of inspiration. They were the result of boring, repeatable systems.

If you're waiting to "feel motivated" to work on your side project after a long day, you'll be waiting forever. Here's what actually works.

Why Motivation Fails and Systems Win

Motivation is an emotion. It peaks after watching a YouTube video of someone who built a SaaS in a weekend and crashes the moment you hit a tricky database migration at 9 PM on a Tuesday. You can't build anything meaningful on emotions alone.

Systems, on the other hand, are decisions you make once and then follow. They remove the need to decide every single day whether you "feel like" working. The decision is already made.

My system is simple:

Two-hour evening blocks, three nights a week (Tuesday, Wednesday, Thursday)
Saturday mornings, 7 AM to 11 AM — my most productive window
Sunday is off. No exceptions. Your brain needs recovery.

The key insight: those two-hour blocks are sacred. I don't check Slack, I don't "quickly" fix a production bug, I don't doom-scroll. The block starts, I open my editor, and I work on the next task. That's it.

Over six months, those blocks add up to roughly 400 hours. That's enough to ship a real product. Not a toy — a real, usable product.

Choosing Projects Strategically

Here's where most developers go wrong: they build what sounds cool instead of what solves a real problem. I've been guilty of this too. I once spent three months building a "revolutionary" task manager that nobody needed, including me.

The best side projects come from your own pain points. You already understand the problem deeply, you're your own first user, and you can validate instantly whether the solution works.

Some of my best ideas came from friction I experienced at work:

A legaltech tool born from watching lawyers manually process documents that could be automated in minutes
An internal automation that saved my team hours of repetitive deployment tasks
A monitoring dashboard because the existing tools were either too expensive or too complex for small teams

The filter I use now:

Does this problem cost me (or someone I know) real time or money?
Can I build a usable V1 in under 8 weeks of side-project time?
Am I genuinely interested enough to use this myself for months?

If any answer is no, I move on. There are infinite ideas. Your time is the scarce resource.

The MVP Mindset for Solo Builders

When you're a team of one, your MVP needs to be ruthlessly small. I'm not talking about the startup pitch deck version of MVP. I mean: what is the absolute minimum thing that delivers value to one person (you)?

For a SaaS I was building to automate contract review, my MVP was:

Upload a PDF
Extract key clauses using a simple NLP pipeline
Display them in a list

No user accounts. No payment system. No beautiful UI. Just the core value proposition working end to end. I built it in three weekends.

Everything else — auth, billing, a proper frontend — came later, after I confirmed the core idea actually worked and people wanted it.

# My MVP checklist before writing a single line of code:
1. Define the ONE thing this does
2. Sketch the happy path (literally on paper)
3. List what I'm NOT building in V1
4. Set a deadline (usually 4-6 weekend sessions)
5. Ship it, even if it's ugly

The "not building" list is more important than the feature list. It keeps you honest when scope creep whispers sweet nothings at 10 PM.

Tech Stack Decisions for Speed

This is where experienced developers actually have a disadvantage. We know too many tools, and we're tempted to pick the "right" one for every situation. For side projects, the right stack is the one you can move fastest with.

My side project stack:

Backend: Go or Node.js with TypeScript — whichever fits the problem better
Frontend: Astro for content sites, Angular or React for apps (whatever I'm most fluent in at the moment)
Database: PostgreSQL. Always. I'm not learning a new database at 9 PM
Hosting: Railway or Fly.io for backends, Vercel or Cloudflare Pages for frontends
Auth: Lucia or an external provider. Never roll your own

The principle: minimize decisions that don't directly relate to your core problem. Every hour spent debating ORMs is an hour not spent on the feature that matters.

I've seen developers spend weeks setting up the "perfect" monorepo with Nx, Turborepo, custom ESLint configs, and CI/CD pipelines — for a project that never shipped a single feature. Don't be that person.

Dealing with Burnout

Let's talk about the elephant in the room. Working full-time and building on the side is taxing. If you don't manage your energy intentionally, you'll burn out. I've been there.

Signs I've learned to recognize:

Dreading the evening block instead of feeling neutral about it
Writing code that I know is bad but not caring enough to fix it
Skipping blocks for two weeks straight and making excuses
Feeling resentful toward the side project

When I notice these signs, I follow a strict protocol:

Take a full week off from the side project. No guilt.
Reassess whether I still care about the problem. If not, kill the project. Seriously.
Reduce the schedule temporarily. Two blocks instead of three. One Saturday morning instead of the full session.
Talk to someone who isn't a developer. My partner's outside perspective has saved me from several pointless rabbit holes.

The goal isn't to hustle until you break. The goal is to build consistently over years. You can't do that if you flame out every three months.

The Compound Effect

Here's what nobody tells you about side projects: the value isn't just the project itself. It's what accumulates around it.

After two years of consistent side project work, I had:

A portfolio of real, shipped products (not tutorial clones)
Deep knowledge in areas outside my day job (legaltech, automation, NLP)
A network of people who found my projects useful
Battle-tested opinions about architecture, because I'd made every mistake myself
Confidence to make technical decisions faster, which directly improved my full-time work

Each project taught me something that made the next one faster. My first SaaS attempt took six months and never launched. My most recent one went from idea to paying users in five weeks. Same number of evening blocks — just compounded experience.

A Closing Thought

Building side projects while working full-time isn't about being a "hustle culture" person. It's about having agency over your growth as an engineer and, potentially, your financial future.

The developers I admire most aren't the ones who grind 80-hour weeks. They're the ones who show up consistently, build things that matter to them, and know when to rest.

Set up your system. Protect your blocks. Ship something small. Then do it again.

The motivation will come and go. The system stays.

From Developer to Architect: The Mindset Shift Nobody Talks About

Thu, 05 Feb 2026 00:00:00 GMT

There's a moment in every senior developer's career where you realize that writing great code isn't enough anymore. The problems you're facing can't be solved with a clever algorithm or a well-placed design pattern. They require you to zoom out — way out — and think about systems as a whole.

That transition from developer to architect broke my brain for a while. Not because it's technically harder, but because it requires you to unlearn habits that made you successful as a developer. Here's what that shift actually looks like.

Thinking in Trade-offs, Not Absolutes

As a developer, I used to think in terms of "the right way" to do things. Normalized databases are better. Microservices are the future. Strong typing prevents bugs. These felt like universal truths.

As an architect, I learned that almost nothing is universally true. Everything is a trade-off, and your job is to pick the trade-off that hurts the least for your specific context.

Should you use microservices? It depends on your team size, deployment capabilities, and how well you understand your domain boundaries. Should you normalize your database? It depends on your read/write ratio and whether you can afford the join complexity at scale.

The "it depends" reflex that junior developers mock is actually hard-earned wisdom. It means you've seen enough systems succeed and fail to know that context determines everything.

Here's a real example. On a project last year, I had to decide between:

Option A: Event-driven architecture with Kafka — technically elegant, great for scale, complex to operate
Option B: Simple REST calls between services with a shared PostgreSQL database — "boring," but the team could ship and maintain it

I chose Option B. The team was four developers, none had Kafka experience, and we had a three-month deadline. Option A would have been architecturally "better" in a vacuum. In our context, it would have been a disaster.

The architect's job isn't to pick the best solution. It's to pick the best solution for this team, this timeline, and this set of constraints.

The Danger of Ivory Tower Architecture

I've worked with architects who design systems from a whiteboard and never look at the code. They produce beautiful diagrams, comprehensive ADRs, and architectures that fall apart the moment developers start implementing them.

The best architects I know — the ones whose systems actually work in production — still read code daily. Not because they don't trust their teams, but because code is where architecture meets reality.

When I review pull requests, I'm not looking for style issues or variable names. I'm looking for:

Are the boundaries I designed actually holding, or are developers working around them?
Is the abstraction I proposed actually reducing complexity, or adding it?
Are there patterns emerging in the code that suggest the architecture needs to evolve?

// This is a smell I look for in code reviews.
// When you see a service directly importing another service's
// internal types, the architectural boundary has been violated.

import (
    "github.com/company/user-service/internal/models"  // RED FLAG
    "github.com/company/order-service/domain"
)

// What it should look like:
import (
    "github.com/company/shared/contracts"  // Shared interface
    "github.com/company/order-service/domain"
)

If your architecture can't survive contact with real developers writing real code under real deadlines, it's not architecture — it's fiction.

Communication Is the Architect's Primary Tool

This one blindsided me. I thought becoming an architect meant spending more time on technical problems. In reality, I spend more time communicating than coding.

An architecture decision that isn't understood by the team is worthless. You need to:

Explain the "why" behind every significant decision. Not just what we're doing, but why this approach over the alternatives
Create shared mental models. When I say "bounded context," every developer on the team should picture the same thing
Listen to pushback. Developers on the ground see things you can't see from the architecture level. If three developers tell you the abstraction is painful, believe them
Write it down. Architecture Decision Records (ADRs) aren't bureaucracy — they're your team's memory

I once designed what I thought was an elegant event sourcing system. During the design review, a mid-level developer asked: "How do we debug this when something goes wrong at 3 AM?" I didn't have a good answer. That question saved us months of pain. The developer didn't need to understand event sourcing theory — they needed to understand operational reality. And they were right.

How DDD Changed My Thinking

Domain-Driven Design was the single biggest influence on my transition to architectural thinking. Not because of the tactical patterns — aggregates, repositories, value objects — but because of the strategic ones.

DDD taught me to:

Start with the domain, not the technology. Before I think about databases or frameworks, I spend time understanding the business problem. What are the entities? What are the processes? Where are the boundaries?
Speak the domain language. If the business calls it a "policy" and your code calls it a "contract," you've created a translation layer that will cause bugs and miscommunication forever.
Identify bounded contexts. This is the most powerful concept in DDD. Not everything needs to share the same model. The "User" in the billing context is different from the "User" in the social context, and that's not just okay — it's correct.

# Before DDD thinking:
One giant User model with 47 fields serving 6 different contexts

# After DDD thinking:
- Billing::Customer (email, payment_method, subscription_tier)
- Social::Profile (display_name, avatar, bio)
- Auth::Identity (credentials, mfa_settings, sessions)
- Support::Requester (name, contact_info, ticket_history)

Each context owns its model. They communicate through well-defined interfaces. Changes in one don't cascade into others. This isn't over-engineering — it's recognizing that a single model serving everyone actually serves no one well.

Making Decisions with Incomplete Information

As a developer, I could gather all the requirements, understand the full scope, and then build. As an architect, I almost never have complete information when I need to make decisions.

You're choosing a messaging system before you know the final scale requirements. You're designing service boundaries before the business model is fully validated. You're picking a cloud provider before you know which regions you'll need.

This used to paralyze me. What if I'm wrong? At this scale, being wrong is expensive.

Here's what I've learned: the cost of not deciding is almost always higher than the cost of a wrong decision. A wrong decision can be corrected. Indecision blocks entire teams.

My framework for decisions under uncertainty:

Make the decision reversible if possible. Use abstractions that let you swap implementations later. This is where ports and adapters architecture shines
Set a review date. "We're going with PostgreSQL for now. In three months, we'll evaluate if we need to move to a distributed database based on actual load data"
Document your assumptions. When you write down "We assume fewer than 10,000 concurrent users in Year 1," you create a tripwire that tells future-you when the decision needs revisiting
Accept that some decisions will be wrong. Design your system so that wrong decisions don't bring everything down. Loose coupling isn't just a buzzword — it's insurance

The Fear of Being Wrong at Scale

When you're a developer and you write a bug, it affects one feature. Maybe a few users see an error message. You fix it, deploy, and move on.

When you're an architect and you make a bad structural decision, it can affect every team, every service, and every deployment for months or years. That weight is real, and pretending it doesn't exist doesn't help.

I deal with it by:

Building proof-of-concepts before committing to major architectural decisions. A two-week spike is cheap compared to a six-month rewrite
Getting input from other experienced engineers. Architecture reviews aren't a sign of weakness — they're a sign of maturity
Studying failures. I learn more from post-mortems than from success stories. When a system fails, the architecture's weaknesses become visible

From "I'll Build It" to "I'll Design It So Others Can Build It"

The hardest part of this transition was letting go of the code. As a developer, my identity was tied to what I built. As an architect, my value comes from enabling others to build well.

This means:

Writing clear specifications instead of writing the code yourself
Trusting your team to make implementation decisions within the boundaries you've set
Measuring success by team output, not personal output
Being okay with solutions that aren't how you would have coded them, as long as they respect the architecture

It's a different kind of satisfaction. Instead of looking at a feature and thinking "I built that," you look at a system and think "I designed the structure that let twelve people build that without stepping on each other."

A Closing Reflection

The transition from developer to architect isn't a promotion — it's a career change. The skills that made you a great developer (deep focus, attention to detail, hands-on problem solving) don't go away, but they become supporting skills rather than primary ones.

Your primary skills become: systems thinking, communication, decision-making under uncertainty, and the humility to know that your beautiful architecture will be bent, broken, and patched by reality.

And honestly? That's what makes it fascinating. You're not building software anymore. You're designing the environment where software gets built. Get that right, and everything else follows.

What Training for a Half Marathon Taught Me About Engineering

Fri, 30 Jan 2026 00:00:00 GMT

The Starting Line Nobody Sees

I signed up for my first half marathon thinking it would be a fun weekend thing. Something to check off a list. What I didn't expect was how deeply the training would mirror the exact challenges I face every day as a Tech Lead building production systems.

I've been doing hybrid training — gym work three days a week combined with running progressions from 10K to 21K — for over a year now. And somewhere around month four, I started noticing that almost every principle that made me a better runner also made me a better engineer. Not in a cute, motivational-poster way. In a real, practical, "this changes how I plan sprints" way.

Let me break it down.

Progressive Overload = Incremental Complexity

In training, progressive overload means you don't jump from running 5K to 21K in a week. You add 10% volume per week. You earn the next level.

In engineering, this is the same principle behind incremental delivery. You don't go from a monolith to microservices in one sprint. You extract one bounded context, validate it works, then extract the next. Every time I've seen a team try to "big bang" a migration, it ended in tears. Every single time.

The body adapts to stress gradually. So does a codebase. So does a team.

When I onboard new engineers, I give them progressively harder tasks — not because I'm gatekeeping, but because context builds on context. You need to understand the data model before you touch the event pipeline. Progressive overload isn't just a training principle. It's a leadership principle.

Rest Days = Preventing Burnout

Here's something gym culture gets wrong and tech culture gets even more wrong: rest is not laziness. Rest is where adaptation happens.

Your muscles don't grow during the workout. They grow during recovery. Your best engineering ideas don't come during a 12-hour coding session. They come in the shower the next morning, after your brain has had time to process.

I enforce rest days on my training schedule with the same discipline I enforce sustainable pace on my teams. No hero sprints. No "let's just push through the weekend." I've seen what happens when you skip rest days in training — you get injured. I've seen what happens when teams skip rest — you get attrition, bugs, and people who stop caring.

Key insight: If your team is consistently working overtime, you don't have a capacity problem. You have a planning problem.

The Training Plan = The Sprint Plan

A good training plan has:

Periodization — cycles of building up and tapering down
Specificity — workouts that target the actual race demands
Measurable milestones — time trials, distance markers
Flexibility — adjustments based on how the body responds

Replace "workouts" with "tasks" and "race" with "release" and you have a solid sprint planning philosophy. The best sprint plans I've run have all of these elements. The worst ones were rigid, unrealistic, and ignored feedback from the team.

I used to plan sprints like a beginner plans training: stuff as much as possible into each week. Now I plan like an experienced runner: leave room for the unexpected, build in recovery, and always know what the priority workout is for that week.

Consistency Beats Intensity

This one changed everything for me.

Running 5K four times a week will make you faster than running 15K once a week and collapsing. Deploying small, well-tested changes daily will get you further than massive releases every two weeks.

In training, consistency builds your aerobic base. In engineering, consistency builds deployment confidence. The teams I lead that ship small and often have fewer incidents, faster recovery times, and — here's the counterintuitive part — more total output than teams that batch everything into big releases.

The same applies to learning. Reading 30 minutes of documentation every day beats a weekend binge. Writing one small PR a day beats a 2,000-line monster on Friday afternoon.

The Wall at KM 18 = The Wall at Month 6

Every distance runner knows "the wall." For a half marathon, it usually hits around kilometer 18. Your glycogen is depleted, your legs are screaming, and your brain is looking for any excuse to stop.

In long-running projects, there's a similar wall. Around month 5 or 6, the initial excitement is gone. The hard architectural problems have been solved, but now you're grinding through integrations, edge cases, and the kind of work nobody celebrates. Team morale dips. People start questioning decisions made in month 1.

How you handle the wall in training:

You expected it. It's part of the plan.
You have nutrition strategy. Gels, hydration, pacing adjustments.
You focus on the next kilometer, not the finish line.

How I handle the wall on projects:

I warn the team early. "Months 5-6 will be tough. That's normal."
I break the remaining work into visible, completable milestones.
I celebrate small wins publicly. Shipped the notification service? That's a win. Say it out loud.

The mental game is real. In both arenas.

Tracking Metrics: Pace/HR = Observability/Monitoring

In running, I track pace, heart rate, cadence, and heart rate variability. Not because I'm obsessed with numbers, but because without data, I'm just guessing whether I'm improving or overtraining.

In production, we track latency, error rates, throughput, and saturation. Same reason.

Here's what both have taught me about metrics:

Vanity metrics are dangerous. Total kilometers run doesn't tell you if you're getting faster. Total deploys doesn't tell you if you're shipping value.
Trends matter more than snapshots. One bad run doesn't mean anything. One spike in error rate doesn't mean anything. But a three-week downward trend in both? Pay attention.
You need baselines. My resting heart rate is 52 bpm. If it jumps to 60, something is wrong — even if 60 is "normal" for someone else. Same with your p99 latency. Know YOUR normal.

// Your observability should tell a story, not just show numbers
// Bad: alert when errors > 100
// Good: alert when error_rate deviates > 2σ from 7-day baseline

alert:
  name: error_rate_anomaly
  condition: |
    abs(current_error_rate - avg_error_rate_7d) > 2 * stddev_error_rate_7d
  severity: warning

Injury Prevention = Tech Debt Management

Runners who skip mobility work, ignore minor pains, and never deload eventually get injured. It's not a question of if, it's when.

Engineers who skip refactoring, ignore code smells, and never pay down tech debt eventually face a production incident caused by accumulated neglect. Same certainty.

The parallel is almost too clean:

Stretching and foam rolling = regular refactoring sessions
Ignoring knee pain = ignoring flaky tests
Running on a stress fracture = shipping on top of known critical bugs
Proper shoes = proper tooling and infrastructure

I schedule "deload weeks" for my teams — sprints where we deliberately reduce feature work and focus on tooling, test coverage, dependency updates, and refactoring. It's not lost productivity. It's injury prevention.

The Mental Game

The hardest part of running 21K isn't physical. By race day, your body is ready. The hard part is the voice in your head at kilometer 15 saying "you could just walk."

The hardest part of leading a long project isn't technical. Your team is skilled enough. The hard part is maintaining conviction when stakeholders are anxious, when the competitor shipped something flashy, when the easy shortcut is right there.

In both cases, the answer is the same: trust the training. Trust the process you built. Trust the plan. Trust that consistency compounds. And when it gets hard — and it will get hard — remember that you prepared for this moment specifically.

Crossing the Finish Line

I finished my first half marathon in 1:52. Not elite. Not record-breaking. But I finished. And I finished because I respected the process, listened to my body, stayed consistent, and didn't quit at kilometer 18.

Every successful project I've led followed the same pattern. Not glamorous. Not revolutionary. Just consistent effort, honest assessment, planned recovery, and the discipline to keep going when it stopped being fun.

The finish line — whether it's a race or a release — is just the visible result of months of invisible work. And that invisible work? That's where the real engineering happens.

Lace up. Ship small. Stay consistent. The wall is coming — and you're ready for it.

Lessons From Running Production Systems With Distributed Teams

Sun, 25 Jan 2026 00:00:00 GMT

The 3 AM Alert That Changed How I Think About Teams

I was deep asleep when the PagerDuty alert hit. An e-commerce checkout service was throwing 500s at a rate that would make your monitoring dashboard look like a seismograph. The service ran on EKS, the queue was backed up in SQS, and our team was spread across three timezones — Ecuador, Argentina, and Spain.

That night taught me more about distributed systems AND distributed teams than any architecture book ever could. Because when production is on fire, it doesn't care where your team lives.

Here's what I've learned from years of running production systems with teams that never share an office.

Observability Is the Great Equalizer

When your team is distributed, nobody can tap a colleague on the shoulder and ask "hey, does this look weird to you?" Observability fills that gap. It becomes the shared language your team speaks across timezones.

We standardized on OpenTelemetry for instrumentation and structured logging everywhere. Not because it's trendy — because when an engineer in Argentina picks up an incident at 9 AM their time, they need to see the SAME context that the engineer in Ecuador saw at 2 AM.

What actually works:

Structured logging with correlation IDs. Every request gets a trace ID that flows through every service, every queue message, every database call. When something breaks, you follow one ID through the entire system. No guessing.
Business-level metrics, not just infra metrics. CPU usage is nice. "Orders per minute dropped 40% in the last 5 minutes" is actionable.
Dashboards that tell stories. We build dashboards around user journeys, not services. The "Checkout Health" dashboard shows everything from cart creation to payment confirmation in one view.

// Structured logging that actually helps during incidents
// Bad: logger.error("Payment failed")
// Good:
logger.error({
  event: 'payment_processing_failed',
  orderId: order.id,
  paymentProvider: 'stripe',
  errorCode: err.code,
  amount: order.total,
  currency: order.currency,
  traceId: context.traceId,
  customerSegment: order.customer.tier,
  retryAttempt: attempt,
  latencyMs: Date.now() - startTime,
});

The difference between those two log lines is the difference between "something broke" and "I know exactly what broke, for whom, and how to fix it."

Incident Management Across Timezones

Incident management with a distributed team requires explicit structure. You can't rely on "everyone jumps on a call" when it's 3 AM for half your team.

Here's the system we built:

Follow-the-sun on-call rotation. Each timezone has primary and secondary on-call. The incident stays with whoever picks it up until it's resolved OR explicitly handed off with a written summary.
Incident channels, not threads. Every P1/P2 gets a dedicated Slack channel with a bot that timestamps every update. When the next timezone wakes up, they read the channel top to bottom and know exactly where things stand.
The 15-minute rule. If you can't identify root cause in 15 minutes, escalate. Not because you're incompetent — because you're managing time, not ego. The sooner you pull in help, the sooner users are unblocked.

The handoff is the hardest part. We use a template:

## Incident Handoff — [INC-2024-0847]
**Status:** Investigating / Mitigated / Monitoring
**Impact:** ~12% of checkout attempts failing in US-East
**What we know:** SQS consumer lag spiking on payment-processor queue
**What we've tried:** Scaled consumers to 8 replicas (no improvement)
**Current hypothesis:** Stripe webhook endpoint is rate-limiting us
**Next steps:** Check Stripe dashboard, consider circuit breaker activation
**Key links:** [Dashboard](link) | [Logs query](link) | [Trace](link)

That template has saved us hours of "wait, what happened while I was asleep?"

Runbooks That Actually Get Used

Most runbooks are graveyard documents. Written once, never updated, never consulted during an actual incident. I've been guilty of this. Here's what changed.

Runbooks work when they're:

Linked directly from alerts. Every PagerDuty alert includes a runbook link. You don't search for it — it's right there in the notification.
Written as decision trees, not novels. "If metric X is above Y, do Z." Not three paragraphs of background context.
Tested regularly. We run "runbook drills" monthly. Pick a scenario, follow the runbook, see if it still works. If it doesn't, update it on the spot.
Owned by the on-call team. The people who use runbooks are the ones who maintain them. Not a separate documentation team.

# runbook: payment-queue-lag
trigger: sqs_queue_depth > 10000 for 5m
steps:
  - check: "Are consumers running?"
    command: "kubectl get pods -l app=payment-consumer -n production"
    if_no: "kubectl rollout restart deployment/payment-consumer -n production"
    if_yes: "continue"
  - check: "Is Stripe responding?"
    command: "curl -o /dev/null -s -w '%{http_code}' https://api.stripe.com/v1/health"
    if_5xx: "Activate circuit breaker: kubectl set env deployment/payment-consumer STRIPE_CIRCUIT=open"
    if_200: "continue"
  - check: "Consumer error logs"
    command: "kubectl logs -l app=payment-consumer --tail=100 | grep ERROR"
    action: "Analyze error pattern and escalate if unknown"

"You Build It, You Run It" — In Practice

This philosophy sounds great in conference talks. In practice, with a distributed team, it requires infrastructure.

You can't expect developers to "run" what they build if:

They don't have access to production logs
They can't deploy without filing a ticket
Alerts go to a separate ops team instead of the building team
There's no way to safely roll back their changes

What we actually implemented:

Every team owns their service's alerts, dashboards, and runbooks. Not ops. Not SRE. The team that writes the code.
Deployment pipelines with automated canary analysis. Engineers deploy their own code. The pipeline watches error rates for 10 minutes. If they spike, it rolls back automatically. This gives developers deployment confidence without requiring an ops gatekeeper.
Feature flags for everything user-facing. Deploy code anytime. Activate features during business hours when the team is awake and watching.

# Canary deployment strategy on EKS
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: checkout-service
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: checkout-service
  progressDeadlineSeconds: 600
  analysis:
    interval: 1m
    threshold: 5
    maxWeight: 50
    stepWeight: 10
    metrics:
      - name: request-success-rate
        thresholdRange:
          min: 99
        interval: 1m
      - name: request-duration
        thresholdRange:
          max: 500
        interval: 1m

Async Communication Patterns That Work

Distributed teams live and die by their async communication. Meetings across timezones are expensive and often unnecessary. Here's what replaced most of ours:

ADRs (Architecture Decision Records) for every significant decision. Not because we love documentation — because when your teammate in Spain wakes up, they need to understand WHY we chose SQS over Kafka, not just that we did.
Video Looms instead of meetings. Five-minute video walkthrough of a PR or a design doc. The receiver watches at 1.5x on their schedule. Response in writing.
RFC documents with explicit deadlines. "Comments due by Thursday EOD UTC." No ambiguity.
Daily async standups in Slack. Three lines: what I did, what I'm doing, any blockers. Posted when your day starts. Read when theirs does.

The rule of thumb: if the information needs to survive longer than the meeting, it should never have been a meeting.

The Deployment Confidence Problem

The scariest moment in distributed team operations is deploying on a Friday afternoon when the team that built the feature is about to go offline for the weekend.

We solved this with layers of confidence:

Feature flags — deploy code dark, activate when the team is online
Canary releases — automated traffic shifting with automatic rollback
Deployment windows — we deploy anytime, but activations happen during overlapping hours (our "golden window" is 10 AM - 1 PM UTC when all timezones are awake)
SNS notifications — every deployment triggers a notification to the team channel with commit summary, author, and rollback instructions

The goal isn't to slow down deployments. It's to make every deployment boring. Boring deployments mean your system is healthy.

War Story: The Silent Queue

One Saturday, our order processing queue (SQS) stopped consuming messages. No errors. No alerts. Just... silence. Orders were piling up, but everything looked green on our dashboards.

The root cause? A Kubernetes node had been cordoned during maintenance, the consumer pods were evicted, and the Horizontal Pod Autoscaler had a minimum replica count of zero. So when the pods were evicted, the HPA was perfectly happy with zero replicas processing zero messages.

What we changed after:

Minimum replica count for critical consumers is now 2. Always.
We added a "zero consumer" alert: if any critical SQS queue has zero active consumers for more than 2 minutes, page immediately.
Runbook updated same day.
Post-mortem shared across all teams, not just the affected one.

No blame. Just learning.

Building a Culture of Ownership, Not Blame

Blameless post-mortems aren't just a nice idea. They're a survival mechanism for distributed teams. Here's why: if people fear blame, they hide problems. If they hide problems, problems grow. If problems grow across timezones, you find out when it's already a disaster.

Our post-mortem template has one rule: no human names in the root cause section. The system failed. Not a person.

We focus on:

What controls were missing that allowed this to happen?
What signals did we miss and why?
What automation would have caught this before users noticed?

The result? Engineers proactively report near-misses. They flag "this feels fragile" without fear. And the whole team gets better because information flows freely.

Documentation Is a Production Dependency

I'll say it plainly: if your system can't be operated by someone who wasn't in the room when it was built, your documentation is failing. And with distributed teams, someone is ALWAYS not in the room.

We treat documentation like we treat tests. It's not optional. It's not "nice to have." It's a production dependency. If you ship a new service without an updated README, operational runbook, and architecture diagram, your PR doesn't get merged.

Is it extra work? Yes. Does it pay for itself the first time someone in a different timezone needs to debug your service at 3 AM? Absolutely.

The Honest Truth

Running production systems with distributed teams is harder than doing it in a co-located office. Anyone who tells you otherwise is selling something. But it's also possible, sustainable, and — when you get it right — incredibly rewarding.

The key isn't fancy tools or complex processes. It's building systems (both technical and human) that assume someone is always sleeping, someone is always debugging, and the documentation is always the first thing they'll read.

Make that experience great, and everything else follows.