GitLab has mastered keeping its massive databases online even during complex schema changes, using smart lock management and strict timeouts. This means users rarely experience downtime from database updates.
Ever wondered how big tech companies update their massive databases without everything grinding to a halt? GitLab has cracked the code, and their approach to zero-downtime database migrations offers valuable lessons for all of us. What this means for you is smoother online experiences, as services like GitLab avoid those annoying 'under maintenance' screens during critical updates.
When a database like GitLab’s handles billions of rows and thousands of transactions every second, even a simple change, like adding a new column, can become a huge problem. Usually, such changes involve what’s called an 'ACCESS EXCLUSIVE' lock, which stops all reading and writing on that part of the database. On a small test database, this is fine – it’s over in milliseconds. But on a gigantic production system, this lock can quickly cause a backlog of transactions, exhaust connections, and ultimately crash your application.
GitLab’s secret sauce involves a deep understanding of how PostgreSQL locks work. They know exactly which operations block others. For instance, while a standard 'SELECT' or 'UPDATE' operation doesn't block much, an 'ALTER TABLE' command with its 'ACCESS EXCLUSIVE' lock will stop everything in its tracks. They’ve also learned that even a quick 2-millisecond 'ALTER TABLE' statement can cause chaos if it gets stuck behind a long-running query. While it waits, every new query trying to access the database gets blocked, and before you know it, the entire application can run out of database connections.
To prevent this, GitLab doesn't allow migrations to wait indefinitely for locks. Instead, they implement strict lock timeouts – as short as 500 milliseconds – and build retry loops into their migration process. This means if a migration can't get the lock quickly, it times out and tries again later, rather than holding up the entire system. By applying these battle-tested patterns, GitLab ensures that essential database changes happen without interrupting your service, setting a high standard for enterprise-level stability.
When a database like GitLab’s handles billions of rows and thousands of transactions every second, even a simple change, like adding a new column, can become a huge problem. Usually, such changes involve what’s called an 'ACCESS EXCLUSIVE' lock, which stops all reading and writing on that part of the database. On a small test database, this is fine – it’s over in milliseconds. But on a gigantic production system, this lock can quickly cause a backlog of transactions, exhaust connections, and ultimately crash your application.
GitLab’s secret sauce involves a deep understanding of how PostgreSQL locks work. They know exactly which operations block others. For instance, while a standard 'SELECT' or 'UPDATE' operation doesn't block much, an 'ALTER TABLE' command with its 'ACCESS EXCLUSIVE' lock will stop everything in its tracks. They’ve also learned that even a quick 2-millisecond 'ALTER TABLE' statement can cause chaos if it gets stuck behind a long-running query. While it waits, every new query trying to access the database gets blocked, and before you know it, the entire application can run out of database connections.
To prevent this, GitLab doesn't allow migrations to wait indefinitely for locks. Instead, they implement strict lock timeouts – as short as 500 milliseconds – and build retry loops into their migration process. This means if a migration can't get the lock quickly, it times out and tries again later, rather than holding up the entire system. By applying these battle-tested patterns, GitLab ensures that essential database changes happen without interrupting your service, setting a high standard for enterprise-level stability.
