Graceful Shutdown

Concept

Graceful shutdown is the process of stopping a service in a way that allows in-flight work to complete before the process exits. Rather than terminating immediately when a signal is received, the service stops accepting new work, finishes what it is currently doing, cleans up resources, and then exits.

Problem

Services are stopped regularly — for deployments, scaling events, or restarts. If a service is terminated abruptly while handling a request, the client receives a connection error. If it is terminated while a database transaction is open, the transaction rolls back but the client has no idea whether the operation succeeded or failed.

In a banking API, an abrupt shutdown during a transfer could leave the client uncertain about whether the money moved. Graceful shutdown eliminates this uncertainty by ensuring requests complete before the process exits.

How it works

In Go, graceful shutdown for an HTTP server follows a predictable pattern:

1. Listen for OS signals (SIGTERM, SIGINT).
2. When a signal arrives, stop accepting new connections.
3. Wait for in-flight requests to complete, up to a timeout.
4. Close other resources (database connections, message queue consumers).
5. Exit.

srv := &http.Server{Addr: ":8080", Handler: router}

// Start server in a goroutine
go func() {
    if err := srv.ListenAndServe(); err != http.ErrServerClosed {
        log.Fatal(err)
    }
}()

// Wait for OS signal
quit := make(chan os.Signal, 1)
signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
<-quit

// Shutdown with a timeout
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()

if err := srv.Shutdown(ctx); err != nil {
    log.Fatal("Server shutdown failed:", err)
}

log.Println("Server stopped cleanly")

srv.Shutdown(ctx) stops the listener, waits for active connections to finish, and returns when they all close or the context deadline passes.

Trade-offs

The shutdown timeout is a design decision. A timeout that is too short risks interrupting long-running requests. A timeout that is too long delays deployments. The right timeout depends on the expected duration of the longest legitimate request. For most APIs, 30 seconds is a reasonable upper bound.

Graceful shutdown requires that in-flight work is actually bounded. If a handler can run for an arbitrary amount of time — because it's waiting on a slow external service — graceful shutdown won't help unless handlers also respect context cancellation.

Practical application

Every HTTP service should implement graceful shutdown. In containerized environments (Docker, Kubernetes), the runtime sends SIGTERM before forcibly killing the process. Without graceful shutdown, the window between SIGTERM and SIGKILL is wasted — active requests are dropped. With graceful shutdown, that window is used to finish in-flight work.

Background workers also need graceful shutdown. A job processor should finish its current job, not abandon it mid-way, when a shutdown signal arrives. The context package and goroutine coordination with sync.WaitGroup are the standard tools for this.

My understanding

Graceful shutdown is one of those things that is easy to skip during initial development — the service works, so why add the complexity? The cost becomes visible during deployments, when every restart potentially drops active requests.

In a banking API, the stakes are higher than in most services. A dropped request mid-transfer puts the client in an unknown state. Graceful shutdown is part of the basic contract of a reliable service: if you send me a request, I will finish handling it or tell you I can't.