Optimistic Locking

Overview

Optimistic locking is a concurrency control mechanism that assumes conflicts between concurrent operations are rare and handles them when they occur, rather than preventing them upfront. It uses version numbers, timestamps, or other metadata to detect when data has been modified by another process, allowing for better performance in low-contention scenarios.

Originally developed for database systems and later popularized by distributed systems and version control systems, optimistic locking has become essential for building high-performance applications. It's widely used at companies like Netflix, Uber, and Airbnb for handling concurrent data modifications without the overhead of traditional locking mechanisms.

Key capabilities include:

• Conflict Detection: Detect concurrent modifications using version numbers
• Performance Optimization: Avoid blocking operations in low-contention scenarios
• Scalability: Better performance than pessimistic locking in distributed systems
• Conflict Resolution: Handle conflicts when they occur

Architecture & Core Components

System Architecture

Core Components

1. Version Management

• Version Numbers: Incremental version tracking for each entity
• Timestamp-based: Use modification timestamps for versioning
• Hash-based: Use content hashes for version detection
• Custom Versions: Application-specific versioning schemes

2. Conflict Detection

• Read-Check-Write: Compare versions before writing
• Atomic Operations: Use database-level atomic operations
• CAS Operations: Compare-and-swap for atomic updates
• Validation Logic: Custom conflict detection rules

3. Conflict Resolution

• Retry Logic: Automatic retry on conflict detection
• Merge Strategies: Intelligent merging of concurrent changes
• User Intervention: Prompt user for conflict resolution
• Last-Writer-Wins: Simple resolution strategy

Locking Flow

Implementation Approaches

1. Database-Level Optimistic Locking

JPA/Hibernate Implementation

@Entity
@Table(name = "users")
public class User {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;
    
    @Column(name = "name")
    private String name;
    
    @Column(name = "email")
    private String email;
    
    @Version
    @Column(name = "version")
    private Long version;
    
    // Constructors, getters, setters
    public User() {}
    
    public User(String name, String email) {
        this.name = name;
        this.email = email;
    }
    
    // Getters and setters
    public Long getId() { return id; }
    public void setId(Long id) { this.id = id; }
    
    public String getName() { return name; }
    public void setName(String name) { this.name = name; }
    
    public String getEmail() { return email; }
    public void setEmail(String email) { this.email = email; }
    
    public Long getVersion() { return version; }
    public void setVersion(Long version) { this.version = version; }
}

@Service
@Transactional
public class UserService {
    
    @Autowired
    private UserRepository userRepository;
    
    public User updateUser(Long id, String name, String email) {
        User user = userRepository.findById(id)
            .orElseThrow(() -> new EntityNotFoundException("User not found"));
        
        user.setName(name);
        user.setEmail(email);
        
        try {
            return userRepository.save(user);
        } catch (OptimisticLockingFailureException e) {
            throw new ConflictException("User was modified by another process");
        }
    }
    
    public User updateUserWithRetry(Long id, String name, String email, int maxRetries) {
        for (int attempt = 0; attempt < maxRetries; attempt++) {
            try {
                return updateUser(id, name, email);
            } catch (ConflictException e) {
                if (attempt == maxRetries - 1) {
                    throw e;
                }
                // Wait before retry
                try {
                    Thread.sleep(100 * (attempt + 1));
                } catch (InterruptedException ie) {
                    Thread.currentThread().interrupt();
                    throw new RuntimeException("Interrupted during retry", ie);
                }
            }
        }
        throw new RuntimeException("Max retries exceeded");
    }
}

2. Application-Level Optimistic Locking

Custom Version Management

public class OptimisticLockingService {
    
    @Autowired
    private RedisTemplate<String, Object> redisTemplate;
    
    @Autowired
    private UserRepository userRepository;
    
    public User updateUserWithOptimisticLocking(Long userId, UserUpdateRequest request) {
        String versionKey = "user:version:" + userId;
        
        // Get current version
        Long currentVersion = (Long) redisTemplate.opsForValue().get(versionKey);
        if (currentVersion == null) {
            // Load from database
            User user = userRepository.findById(userId)
                .orElseThrow(() -> new EntityNotFoundException("User not found"));
            currentVersion = user.getVersion();
            redisTemplate.opsForValue().set(versionKey, currentVersion, Duration.ofMinutes(30));
        }
        
        // Check if version matches
        if (!currentVersion.equals(request.getExpectedVersion())) {
            throw new ConflictException("Version mismatch. Expected: " + request.getExpectedVersion() + 
                                      ", Current: " + currentVersion);
        }
        
        // Perform update
        User updatedUser = userRepository.save(request.toUser());
        
        // Update version in cache
        redisTemplate.opsForValue().set(versionKey, updatedUser.getVersion(), Duration.ofMinutes(30));
        
        return updatedUser;
    }
}

Performance Characteristics

Concurrency Metrics

Conflict Rates

• Low Contention: < 5% conflict rate, optimal performance
• Medium Contention: 5-20% conflict rate, acceptable performance
• High Contention: > 20% conflict rate, consider pessimistic locking
• Burst Contention: Temporary spikes in conflict rates

Performance Comparison

• Optimistic Locking: Better performance in low-contention scenarios
• Pessimistic Locking: Better performance in high-contention scenarios
• No Locking: Best performance but risk of data corruption
• Hybrid Approach: Best of both worlds with complexity

Latency Impact

Read Operations

• Version Retrieval: Minimal overhead (1-2ms)
• Cache Lookup: Sub-millisecond for cached versions
• Database Query: Additional version column in SELECT
• Network Overhead: Minimal for version metadata

Write Operations

• Version Check: Additional WHERE clause in UPDATE
• Conflict Detection: Immediate failure on version mismatch
• Retry Logic: Exponential backoff for retries
• Success Path: Same performance as regular UPDATE

Production Best Practices

Configuration Guidelines

Version Strategy Selection

public enum VersionStrategy {
    INCREMENTAL("Incremental version numbers", "1, 2, 3, 4..."),
    TIMESTAMP("Timestamp-based versioning", "2023-01-01T10:00:00Z"),
    HASH("Content hash versioning", "SHA-256 hash of content"),
    CUSTOM("Application-specific versioning", "Custom format");
    
    private final String description;
    private final String example;
    
    VersionStrategy(String description, String example) {
        this.description = description;
        this.example = example;
    }
}

Conflict Resolution Strategies

public class ConflictResolutionStrategies {
    
    // 1. Automatic Retry
    public User updateWithRetry(Long userId, UserUpdateRequest request) {
        for (int attempt = 0; attempt < maxRetries; attempt++) {
            try {
                return updateUser(userId, request);
            } catch (ConflictException e) {
                if (attempt == maxRetries - 1) {
                    throw e;
                }
                // Exponential backoff
                try {
                    Thread.sleep(retryDelayMs * (1L << attempt));
                } catch (InterruptedException ie) {
                    Thread.currentThread().interrupt();
                    throw new RuntimeException("Interrupted during retry", ie);
                }
            }
        }
        throw new RuntimeException("Max retries exceeded");
    }
    
    // 2. Last Writer Wins
    public User updateWithLWW(Long userId, UserUpdateRequest request) {
        User user = userRepository.findById(userId)
            .orElseThrow(() -> new EntityNotFoundException("User not found"));
        
        // Always update, ignoring version conflicts
        user.setName(request.getName());
        user.setEmail(request.getEmail());
        user.setVersion(user.getVersion() + 1);
        
        return userRepository.save(user);
    }
}

Monitoring and Observability

Key Metrics to Track

@Component
public class OptimisticLockingMetrics {
    
    private final MeterRegistry meterRegistry;
    private final Counter conflictCounter;
    private final Timer updateTimer;
    private final Gauge conflictRateGauge;
    
    public OptimisticLockingMetrics(MeterRegistry meterRegistry) {
        this.meterRegistry = meterRegistry;
        this.conflictCounter = Counter.builder("optimistic.locking.conflicts")
            .description("Number of optimistic locking conflicts")
            .register(meterRegistry);
        this.updateTimer = Timer.builder("optimistic.locking.update.duration")
            .description("Time taken for optimistic locking updates")
            .register(meterRegistry);
        this.conflictRateGauge = Gauge.builder("optimistic.locking.conflict.rate")
            .description("Current conflict rate percentage")
            .register(meterRegistry, this, OptimisticLockingMetrics::getConflictRate);
    }
    
    public void recordConflict() {
        conflictCounter.increment();
    }
    
    public void recordUpdate(Duration duration) {
        updateTimer.record(duration);
    }
    
    public double getConflictRate() {
        // Calculate conflict rate based on recent metrics
        return calculateRecentConflictRate();
    }
    
    private double calculateRecentConflictRate() {
        // Implementation to calculate conflict rate
        return 0.0; // Placeholder
    }
}

Health Checks

@Component
public class OptimisticLockingHealthIndicator implements HealthIndicator {
    
    @Autowired
    private OptimisticLockingMetrics metrics;
    
    @Override
    public Health health() {
        double conflictRate = metrics.getConflictRate();
        
        if (conflictRate > 20.0) {
            return Health.down()
                .withDetail("conflictRate", conflictRate)
                .withDetail("status", "High conflict rate detected")
                .build();
        } else if (conflictRate > 10.0) {
            return Health.up()
                .withDetail("conflictRate", conflictRate)
                .withDetail("status", "Elevated conflict rate")
                .build();
        } else {
            return Health.up()
                .withDetail("conflictRate", conflictRate)
                .withDetail("status", "Normal operation")
                .build();
        }
    }
}

Interview Questions

Basic Level

Q1: What is optimistic locking and how does it differ from pessimistic locking?

Answer: Optimistic locking assumes that conflicts are rare and allows multiple transactions to proceed concurrently. It detects conflicts at commit time by checking if the data has been modified since it was read. Pessimistic locking prevents conflicts by acquiring locks before accessing data, blocking other transactions until the lock is released.

Q2: What are the main components of optimistic locking?

Answer:

• Version Field: Tracks the current version of the data
• Version Check: Compares expected version with current version
• Conflict Detection: Identifies when versions don't match
• Conflict Resolution: Handles conflicts through retry, merge, or error

Q3: When should you use optimistic locking vs pessimistic locking?

Answer: Use optimistic locking when:

• Conflicts are infrequent (< 20% of operations)
• Read operations are more common than writes
• You need high concurrency and performance
• You can handle occasional conflicts

Use pessimistic locking when:

• Conflicts are frequent (> 20% of operations)
• Data consistency is critical
• You can't afford retry overhead
• You need guaranteed exclusive access

Intermediate Level

Q4: How do you implement optimistic locking in a distributed system?

Answer:

// Distributed optimistic locking with Redis
public class DistributedOptimisticLocking {
    
    @Autowired
    private RedisTemplate<String, Object> redisTemplate;
    
    public boolean acquireOptimisticLock(String resourceId, String clientId, long timeoutMs) {
        String lockKey = "lock:" + resourceId;
        String versionKey = "version:" + resourceId;
        
        // Get current version
        Long currentVersion = (Long) redisTemplate.opsForValue().get(versionKey);
        if (currentVersion == null) {
            currentVersion = 0L;
        }
        
        // Set lock with expiration
        Boolean acquired = redisTemplate.opsForValue()
            .setIfAbsent(lockKey, clientId, Duration.ofMillis(timeoutMs));
        
        if (acquired) {
            // Store expected version for later validation
            redisTemplate.opsForValue().set("expected:" + resourceId, currentVersion);
            return true;
        }
        
        return false;
    }
    
    public boolean releaseOptimisticLock(String resourceId, String clientId, Long expectedVersion) {
        String lockKey = "lock:" + resourceId;
        String versionKey = "version:" + resourceId;
        String expectedKey = "expected:" + resourceId;
        
        // Verify lock ownership
        String lockOwner = (String) redisTemplate.opsForValue().get(lockKey);
        if (!clientId.equals(lockOwner)) {
            return false;
        }
        
        // Check version consistency
        Long currentVersion = (Long) redisTemplate.opsForValue().get(versionKey);
        Long storedExpected = (Long) redisTemplate.opsForValue().get(expectedKey);
        
        if (!expectedVersion.equals(currentVersion) || !expectedVersion.equals(storedExpected)) {
            // Version conflict detected
            redisTemplate.delete(lockKey);
            redisTemplate.delete(expectedKey);
            return false;
        }
        
        // Release lock and update version
        redisTemplate.delete(lockKey);
        redisTemplate.delete(expectedKey);
        redisTemplate.opsForValue().increment(versionKey);
        
        return true;
    }
}

Q5: What are the performance implications of optimistic locking?

Answer:Read Performance:

• Minimal overhead for version retrieval
• Additional column in SELECT queries
• Cache-friendly for version metadata

Write Performance:

• Additional WHERE clause in UPDATE statements
• Immediate failure on conflicts (no blocking)
• Retry overhead for conflict resolution
• Success path has same performance as regular updates

Memory Usage:

• Version fields consume minimal storage
• Cache overhead for version metadata
• Retry state management

Network Overhead:

• Version metadata in requests/responses
• Additional round trips for conflict resolution

Advanced Level

Q6: How would you handle optimistic locking in a microservices architecture?

Answer:

// Saga pattern with optimistic locking
public class OptimisticLockingSaga {
    
    @Autowired
    private SagaOrchestrator sagaOrchestrator;
    
    @Autowired
    private OptimisticLockingService lockingService;
    
    public void executeOrderSaga(OrderRequest request) {
        SagaContext context = new SagaContext();
        
        try {
            // Step 1: Reserve inventory with optimistic locking
            InventoryReservation reservation = reserveInventory(request.getItems(), context);
            context.addStep("inventory", reservation);
            
            // Step 2: Process payment with optimistic locking
            PaymentResult payment = processPayment(request.getPayment(), context);
            context.addStep("payment", payment);
            
            // Step 3: Create order with optimistic locking
            Order order = createOrder(request, context);
            context.addStep("order", order);
            
            // Commit all changes
            commitSaga(context);
            
        } catch (ConflictException e) {
            // Handle conflicts and retry
            handleSagaConflict(context, e);
        } catch (Exception e) {
            // Compensate for partial execution
            compensateSaga(context, e);
        }
    }
    
    private void handleSagaConflict(SagaContext context, ConflictException e) {
        // Implement conflict resolution strategy
        if (context.getRetryCount() < maxRetries) {
            // Wait and retry
            try {
                Thread.sleep(calculateBackoffDelay(context.getRetryCount()));
                context.incrementRetryCount();
                executeOrderSaga(context.getOriginalRequest());
            } catch (InterruptedException ie) {
                Thread.currentThread().interrupt();
                compensateSaga(context, ie);
            }
        } else {
            // Max retries exceeded, compensate
            compensateSaga(context, e);
        }
    }
}

Q7: How do you prevent the ABA problem in optimistic locking?

Answer: The ABA problem occurs when a value changes from A to B and back to A, making it appear unchanged. Solutions:

// 1. Use version numbers instead of values
public class VersionedEntity {
    private Long id;
    private String data;
    private Long version; // Increments on every change
    
    // Version always changes, even if data returns to original value
}

// 2. Use timestamps with sufficient precision
public class TimestampedEntity {
    private Long id;
    private String data;
    private Instant lastModified; // High precision timestamp
    
    // Timestamps are monotonically increasing
}

// 3. Use content hashes
public class HashedEntity {
    private Long id;
    private String data;
    private String contentHash; // SHA-256 of data + metadata
    
    // Hash changes with any modification
}

// 4. Combine multiple approaches
public class RobustOptimisticLocking {
    private Long version;
    private Instant timestamp;
    private String contentHash;
    private String clientId;
    
    // Multiple validation layers prevent ABA
    public boolean validateConsistency(RobustOptimisticLocking expected) {
        return this.version.equals(expected.version) &&
               this.timestamp.equals(expected.timestamp) &&
               this.contentHash.equals(expected.contentHash) &&
               this.clientId.equals(expected.clientId);
    }
}

Real-World Scenarios

Scenario 1: E-commerce Inventory Management

Problem: Multiple users trying to purchase the last item in stock simultaneously.

Solution: Optimistic locking with inventory versioning.

@Service
public class InventoryService {
    
    public PurchaseResult purchaseItem(Long itemId, int quantity, String userId) {
        for (int attempt = 0; attempt < maxRetries; attempt++) {
            try {
                // Get current inventory with version
                InventoryItem item = inventoryRepository.findById(itemId)
                    .orElseThrow(() -> new ItemNotFoundException("Item not found"));
                
                // Check availability
                if (item.getStock() < quantity) {
                    return PurchaseResult.outOfStock();
                }
                
                // Create purchase with version check
                Purchase purchase = new Purchase(itemId, quantity, userId, item.getVersion());
                
                // Update inventory with optimistic locking
                item.setStock(item.getStock() - quantity);
                item.setVersion(item.getVersion() + 1);
                
                inventoryRepository.save(item);
                purchaseRepository.save(purchase);
                
                return PurchaseResult.success(purchase);
                
            } catch (OptimisticLockingFailureException e) {
                if (attempt == maxRetries - 1) {
                    return PurchaseResult.conflict("Unable to complete purchase due to high demand");
                }
                // Wait and retry
                try {
                    Thread.sleep(100 * (attempt + 1));
                } catch (InterruptedException ie) {
                    Thread.currentThread().interrupt();
                    return PurchaseResult.error("Purchase interrupted");
                }
            }
        }
        return PurchaseResult.error("Max retries exceeded");
    }
}

Scenario 2: Collaborative Document Editing

Problem: Multiple users editing the same document simultaneously.

Solution: Optimistic locking with operational transformation.

@Service
public class DocumentEditingService {
    
    public EditResult applyEdit(Long documentId, DocumentEdit edit, String userId) {
        try {
            // Get document with version
            Document document = documentRepository.findById(documentId)
                .orElseThrow(() -> new DocumentNotFoundException("Document not found"));
            
            // Check if edit is compatible with current version
            if (!edit.getExpectedVersion().equals(document.getVersion())) {
                // Get conflicting edits since expected version
                List<DocumentEdit> conflictingEdits = getConflictingEdits(documentId, edit.getExpectedVersion());
                
                // Transform edit to resolve conflicts
                DocumentEdit transformedEdit = transformEdit(edit, conflictingEdits);
                
                // Apply transformed edit
                document.applyEdit(transformedEdit);
                document.setVersion(document.getVersion() + 1);
                
                documentRepository.save(document);
                
                return EditResult.success(transformedEdit);
            } else {
                // No conflicts, apply edit directly
                document.applyEdit(edit);
                document.setVersion(document.getVersion() + 1);
                
                documentRepository.save(document);
                
                return EditResult.success(edit);
            }
            
        } catch (ConflictException e) {
            return EditResult.conflict("Edit conflicts with other changes");
        }
    }
    
    private DocumentEdit transformEdit(DocumentEdit originalEdit, List<DocumentEdit> conflictingEdits) {
        // Implement operational transformation logic
        // This is a simplified version - real implementation would be more complex
        DocumentEdit transformed = originalEdit;
        
        for (DocumentEdit conflict : conflictingEdits) {
            transformed = transformEditAgainst(transformed, conflict);
        }
        
        return transformed;
    }
}

Scenario 3: Financial Transaction Processing

Problem: Concurrent transactions on the same account causing balance inconsistencies.

Solution: Optimistic locking with account versioning and transaction isolation.

@Service
@Transactional
public class AccountService {
    
    public TransactionResult transferMoney(Long fromAccountId, Long toAccountId, 
                                         BigDecimal amount, String transactionId) {
        try {
            // Get both accounts with versions
            Account fromAccount = accountRepository.findById(fromAccountId)
                .orElseThrow(() -> new AccountNotFoundException("From account not found"));
            Account toAccount = accountRepository.findById(toAccountId)
                .orElseThrow(() -> new AccountNotFoundException("To account not found"));
            
            // Validate sufficient balance
            if (fromAccount.getBalance().compareTo(amount) < 0) {
                return TransactionResult.insufficientFunds();
            }
            
            // Create transaction record
            Transaction transaction = new Transaction(transactionId, fromAccountId, toAccountId, amount);
            
            // Update accounts with optimistic locking
            fromAccount.setBalance(fromAccount.getBalance().subtract(amount));
            fromAccount.setVersion(fromAccount.getVersion() + 1);
            
            toAccount.setBalance(toAccount.getBalance().add(amount));
            toAccount.setVersion(toAccount.getVersion() + 1);
            
            // Save all changes atomically
            accountRepository.save(fromAccount);
            accountRepository.save(toAccount);
            transactionRepository.save(transaction);
            
            return TransactionResult.success(transaction);
            
        } catch (OptimisticLockingFailureException e) {
            // Handle concurrent modification
            return TransactionResult.conflict("Account was modified by another transaction");
        } catch (Exception e) {
            // Handle other errors
            return TransactionResult.error("Transaction failed: " + e.getMessage());
        }
    }
}

Conclusion

Optimistic locking is a powerful concurrency control mechanism that provides high performance and scalability for applications with low to moderate contention. By detecting conflicts at commit time rather than preventing them upfront, optimistic locking enables better resource utilization and user experience.

The key to successful optimistic locking implementation is:

1. Proper Version Management: Choose appropriate versioning strategies
2. Conflict Resolution: Implement robust retry and resolution mechanisms
3. Performance Monitoring: Track conflict rates and adjust strategies
4. Error Handling: Graceful degradation when conflicts occur
5. Testing: Comprehensive testing of conflict scenarios

When implemented correctly, optimistic locking can significantly improve application performance while maintaining data consistency in distributed systems.