dotcom-arcade/docs/08-development/database-migrations.md

Database Migrations

This document describes the database migration system for the user database (user.db).

Overview

The migration system uses a registry-based approach to track which migrations have been applied to the database. This ensures consistent schema evolution across all installations.

Architecture

Migration Registry

All applied migrations are tracked in the migrations table:

CREATE TABLE migrations (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  name TEXT NOT NULL UNIQUE,
  applied_at TEXT NOT NULL,
  checksum TEXT,
  execution_time_ms INTEGER,
  description TEXT
);

Migration Files

Migration files are located in backend/src/migrations/ and follow the naming convention:

XXX_description.sql

• XXX - Three-digit version number (001, 002, 003, etc.)
• description - Brief kebab-case description of the migration

Example: 001_initialize_schema.sql

Current Active Migrations

File	Description
bootstrap.sql	Creates the migrations registry table
001_initialize_schema.sql	Complete database schema with all tables
002_seed_default_data.sql	Default roles, permissions, and system settings

Archived Migrations

Historical migration files (001-016 from the old system) are archived in backend/src/migrations/archived/ for reference. These files are no longer executed but document the evolution of the database schema.

How It Works

Startup Flow

When the backend starts, UserDatabaseService.initialize() performs these steps:

1. Bootstrap - Creates the migrations table if it doesn't exist
2. Detect Existing State - For backward compatibility, detects if the database already has tables and marks migrations as applied
3. Run Migrations - Scans the migrations directory and runs any migrations not yet in the registry
4. Record Execution - Logs each migration with timestamp, checksum, and execution time

Backward Compatibility

The system automatically detects existing databases (from the old migration system) and marks the consolidated migrations as applied:

// Checks for existing tables
if (tables.some(t => t.name === 'users')) {
  markApplied('001_initialize_schema', 'Detected existing schema');
}

// Checks for existing seed data
const hasRoles = this.db.prepare('SELECT COUNT(*) as count FROM roles').get();
if (hasRoles && hasRoles.count > 0) {
  markApplied('002_seed_default_data', 'Detected existing seed data');
}

This ensures a smooth transition from the old ad-hoc migration system to the new registry-based system.

Adding New Migrations

Step 1: Create Migration File

Create a new SQL file in backend/src/migrations/ with the next sequential number:

# Example: Adding a new feature
touch backend/src/migrations/003_add_feature_x.sql

Step 2: Write Idempotent SQL

All migrations must be idempotent (safe to run multiple times):

-- Good: Idempotent operations
CREATE TABLE IF NOT EXISTS my_table (...);
CREATE INDEX IF NOT EXISTS idx_name ON my_table(column);
INSERT OR IGNORE INTO settings (key, value) VALUES ('key', 'value');

-- Bad: Non-idempotent operations
CREATE TABLE my_table (...);  -- Fails if table exists
INSERT INTO settings (key, value) VALUES ('key', 'value');  -- Fails if key exists

Step 3: Add Comments

Include migration metadata and rollback instructions:

-- Migration: 003_add_feature_x.sql
-- Description: Add feature X tables and settings
-- Date: 2026-XX-XX
-- Rollback: DROP TABLE feature_x; DELETE FROM settings WHERE key LIKE 'feature_x.%';

-- Your migration SQL here...

Step 4: Test Migration

Test on a fresh database and an existing database:

# Fresh database test
rm backend/user.db
npm run dev:backend  # Check logs for migration execution

# Existing database test
npm run dev:backend  # Should detect migration as new and apply it

Step 5: Verify Registry

After the migration runs, verify it was recorded:

SELECT * FROM migrations ORDER BY applied_at DESC LIMIT 1;

Best Practices

1. Idempotency is Required

All migrations must be safe to run multiple times:

• Use IF NOT EXISTS for table/index creation
• Use INSERT OR IGNORE for seed data with unique constraints
• Use INSERT OR REPLACE for settings that should update
• Avoid raw UPDATE statements without conditions

2. Never Modify Applied Migrations

Once a migration has been applied to any database:

• Never change the SQL content
• Never rename the file
• Never delete the file

If you need to make changes, create a new migration that modifies the schema.

3. Use Descriptive Names

Migration names should clearly indicate what they do:

✅ Good:

• 003_add_user_badges.sql
• 004_remove_deprecated_settings.sql
• 005_add_game_tags_index.sql

❌ Bad:

• 003_changes.sql
• 004_fix.sql
• 005_update.sql

4. Keep Migrations Focused

Each migration should have a single, clear purpose:

• ✅ One feature or change per migration
• ❌ Don't bundle multiple unrelated changes

5. Test on Fresh and Existing Databases

Always test migrations on:

1. Fresh database - Ensure it works from scratch
2. Existing database - Ensure it applies correctly to production-like state

6. Document Rollback Procedures

Include rollback instructions in comments:

-- Rollback: DROP TABLE new_table; DELETE FROM system_settings WHERE key = 'new_feature.enabled';

For complex migrations, create a separate rollback file:

migrations/
  003_add_feature_x.sql
  rollback/
    003_add_feature_x_rollback.sql

7. Use Transactions for Complex Changes

For migrations that modify data:

BEGIN TRANSACTION;

-- Your migration SQL here...

COMMIT;

SQLite supports transactions for schema changes, so use them when appropriate.

Migration File Structure

Schema Migrations

For creating tables, indexes, and constraints:

-- Migration: XXX_description.sql
-- Description: Brief description
-- Date: YYYY-MM-DD
-- Rollback: Instructions here

-- ===================================
-- TABLE: table_name
-- ===================================
CREATE TABLE IF NOT EXISTS table_name (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  name TEXT NOT NULL,
  created_at TEXT DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now'))
);

CREATE INDEX IF NOT EXISTS idx_table_name ON table_name(name);

Data Migrations

For inserting or updating data:

-- Migration: XXX_seed_data.sql
-- Description: Insert default configuration
-- Date: YYYY-MM-DD

INSERT OR IGNORE INTO system_settings (key, value, data_type, category, description)
VALUES
  ('feature.enabled', '1', 'boolean', 'features', 'Enable feature X'),
  ('feature.setting', 'default', 'string', 'features', 'Feature X setting');

Transformation Migrations

For modifying existing data:

-- Migration: XXX_transform_data.sql
-- Description: Convert format of existing data
-- Date: YYYY-MM-DD
-- Rollback: Manual restore required

UPDATE table_name
SET
  field = CASE
    WHEN condition1 THEN value1
    WHEN condition2 THEN value2
    ELSE default_value
  END
WHERE field NOT LIKE 'new_format%';  -- Idempotent check

Troubleshooting

Migration Fails on Existing Database

Symptom: Migration runs successfully on fresh database but fails on existing database.

Cause: Non-idempotent SQL (e.g., CREATE TABLE without IF NOT EXISTS)

Solution: Add idempotency checks:

CREATE TABLE IF NOT EXISTS ...
INSERT OR IGNORE INTO ...

Migration Shows as Not Applied

Symptom: Migration is in the file system but not in migrations table.

Diagnosis:

-- Check what migrations are registered
SELECT name, applied_at FROM migrations ORDER BY applied_at;

-- Check file system
ls backend/src/migrations/*.sql

Solution: The migration may not have run yet. Restart the backend to trigger migration execution.

Checksum Mismatch

Symptom: Migration was applied but someone modified the SQL file.

Detection:

SELECT name, checksum FROM migrations WHERE name = '003_my_migration';

Then compute the checksum of the current file and compare.

Solution:

1. If the change was intentional, create a new migration with the corrected SQL
2. If the change was accidental, restore the original SQL from git history
3. Never modify applied migrations

Database Locked Error

Symptom: SQLITE_BUSY or database locked errors during migration.

Cause: Another process (e.g., Flashpoint Launcher) has the database locked.

Solution:

1. Close the Flashpoint Launcher
2. Ensure no other backend processes are running
3. Restart the backend

Viewing Migration History

Query Migration Registry

-- View all applied migrations
SELECT
  name,
  applied_at,
  execution_time_ms,
  description
FROM migrations
ORDER BY applied_at DESC;

-- View most recent migration
SELECT * FROM migrations ORDER BY applied_at DESC LIMIT 1;

Check Migration Status

# View migration logs
grep "Migration completed" logs/app.log

# View migration execution times
sqlite3 backend/user.db "SELECT name, execution_time_ms FROM migrations ORDER BY applied_at DESC;"

Recovery Procedures

Restore from Backup

If a migration fails and corrupts the database:

1. Stop the backend
2. Restore user.db from backup
3. Remove the failed migration file
4. Fix the migration SQL
5. Re-add the migration file with corrected SQL
6. Restart the backend

Manual Migration Rollback

If you need to roll back a migration:

1. Stop the backend
2. Run the rollback SQL (see migration comments)
3. Remove the migration record from the registry:

   DELETE FROM migrations WHERE name = 'XXX_migration_name';
   

4. Restart the backend

Reset Migration Registry

Warning: This should only be done on development databases, never in production.

-- Delete all migration records
DELETE FROM migrations;

-- Or drop the entire table
DROP TABLE migrations;

-- Restart backend to re-bootstrap

Comparison: Old vs New System

Old System (Before 2026-01-28)

• ❌ No migration tracking table
• ❌ Ad-hoc detection logic in TypeScript
• ❌ Difficult to determine database state
• ❌ 16 separate migration files
• ❌ Complex conditional logic for each migration

New System (Registry-Based)

• ✅ Formal migration registry table
• ✅ Simple, consistent detection logic
• ✅ Easy to query database state
• ✅ 2 consolidated migration files (001, 002)
• ✅ Clean, maintainable code

References

• Migration Registry Implementation: backend/src/services/UserDatabaseService.ts
• Migration Files: backend/src/migrations/
• Archived Migrations: backend/src/migrations/archived/
• Database Schema Reference: docs/12-reference/database-schema-reference.md