Endrias Bridge
Windows Guest Initialization Tool — automates hostname, user, password, networking, SSH keys, and userdata script execution at VM boot time.
What is Endrias Bridge?
Endrias Bridge is the Windows equivalent of Cloud-Init (Linux). When a Windows virtual machine boots for the first time on any hypervisor (Hyper-V, KVM, VMware, Xen) or cloud platform (OpenStack, AWS, Azure, Oracle Cloud), Endrias Bridge reads metadata from the environment and automatically configures the guest OS — no manual intervention required.
| Capability | What it does | Source |
|---|---|---|
| 🖥 Hostname | Renames the computer and schedules a reboot | Metadata |
| 👤 User Creation | Creates a local Windows account + group membership | Config / Metadata |
| 🔑 Password | Injects metadata password or generates a secure random one | Metadata / Generated |
| 🌐 Networking | Configures static IP, gateway, DNS via PowerShell | Metadata |
| 🗝 SSH Keys | Writes public keys to authorized_keys | Metadata |
| 📜 Userdata | Executes PowerShell, CMD, or Bash scripts | Metadata |
Architecture
| File / Folder | Purpose |
|---|---|
migrationbridge.conf | Main INI config file — edit to customise behaviour |
migrationbridge\config.py | Config singleton with all defaults |
migrationbridge\init_manager.py | Boot orchestrator — loads sources and runs plugins |
migrationbridge\service.py | Windows Service wrapper (pywin32) |
migrationbridge\metadata\ | One file per metadata source |
migrationbridge\plugins\ | One file per initialization plugin |
migrationbridge\utils\ | PowerShell helpers and network utilities |
Check Service Status
Run in any PowerShell window (no admin needed to query):
Quick one-liner
Get-Service "Endrias Bridge"
Expected output when healthy:
------ ---- -----------
Running Endrias Bridge Endrias Bridge Guest Init Service
Detailed status with start type
Get-Service "Endrias Bridge" | Select-Object Name, Status, StartType, DisplayName
| Status value | Meaning | Action |
|---|---|---|
| Running | Service is active | All good — check log for plugin results |
| Stopped | Finished and stopped itself (normal if stop_service_on_exit=true) | Check log — this is expected after first boot |
| StartPending | Still initialising | Wait 30 s then re-check |
| StopPending / Error | Failed to start | Check Windows Event Log (see § Event Log) |
stop_service_on_exit = true so the service stops itself once all plugins finish. The VM is fully configured at that point.Read the Logs
The default log file is C:\Windows\Temp\migrationbridge.log.
Tail the live log
Get-Content C:\Windows\Temp\migrationbridge.log -Wait -Tail 40
Read the entire log
Get-Content C:\Windows\Temp\migrationbridge.log
Filter for errors only
Select-String -Path C:\Windows\Temp\migrationbridge.log -Pattern 'ERROR|WARNING'
Sample healthy log output
2026-06-05 08:00:01 INFO migrationbridge.init_manager: First boot detected
2026-06-05 08:00:01 INFO migrationbridge.metadata.configdrive: Loaded metadata from ConfigDrive at D:\openstack\latest
2026-06-05 08:00:01 INFO migrationbridge.init_manager: Active metadata source: ...configdrive.ConfigDriveSource
2026-06-05 08:00:01 INFO migrationbridge.init_manager: Running plugin: SetHostnamePlugin
2026-06-05 08:00:02 INFO migrationbridge.plugins.set_hostname: Hostname set to WIN-WEBSERVER01 — reboot required to take effect
2026-06-05 08:00:02 INFO migrationbridge.init_manager: Running plugin: CreateUserPlugin
2026-06-05 08:00:02 INFO migrationbridge.plugins.create_user: Created user: Admin
2026-06-05 08:00:02 INFO migrationbridge.plugins.create_user: Added Admin to group: Administrators
2026-06-05 08:00:02 INFO migrationbridge.init_manager: Running plugin: SetPasswordPlugin
2026-06-05 08:00:02 INFO migrationbridge.plugins.set_password: Password configured for user Admin
2026-06-05 08:00:03 INFO migrationbridge.init_manager: Running plugin: NetworkConfigPlugin
2026-06-05 08:00:03 INFO migrationbridge.plugins.network_config: Configured adapter Ethernet: 10.0.0.5/255.255.255.0 gw=10.0.0.1
2026-06-05 08:00:03 INFO migrationbridge.init_manager: Running plugin: InjectSSHKeysPlugin
2026-06-05 08:00:03 INFO migrationbridge.plugins.inject_ssh_keys: Injected 2 SSH key(s) for user Admin
2026-06-05 08:00:04 INFO migrationbridge.init_manager: Running plugin: ExecuteUserdataPlugin
2026-06-05 08:00:05 INFO migrationbridge.plugins.execute_userdata: Userdata script completed successfully
2026-06-05 08:00:05 INFO migrationbridge.init_manager: Initiating system reboot as requested by plugin(s)
First-Boot Marker
Endrias Bridge writes a marker file after completing its first boot run. This prevents first-boot-only tasks from repeating on every restart.
Check if first boot has run
Test-Path C:\Windows\Temp\migrationbridge_firstboot # True → first boot already completed # False → first boot has NOT run yet
Force a re-run (simulate fresh deployment)
Remove-Item C:\Windows\Temp\migrationbridge_firstboot -Force Restart-Service "Endrias Bridge"
Verify Plugin Results
After Endrias Bridge runs, use these commands to confirm each plugin did its job.
Hostname
hostname # or $env:COMPUTERNAME
User was created
net user Admin # Look for "Account active: Yes" and correct group membership
User group membership
net localgroup Administrators
SSH keys were injected
Get-Content C:\Users\Admin\.ssh\authorized_keys
Network configuration
Get-NetIPAddress -AddressFamily IPv4 | Format-Table InterfaceAlias, IPAddress, PrefixLength Get-NetRoute -DestinationPrefix 0.0.0.0/0 Get-DnsClientServerAddress
Service Management
All commands below require an elevated (Administrator) PowerShell.
| Action | Command |
|---|---|
| Install service | migrationbridge service install |
| Start service | migrationbridge service start |
| Stop service | migrationbridge service stop |
| Remove service | migrationbridge service uninstall |
| Restart service | Restart-Service "Endrias Bridge" |
| Set auto-start | Set-Service "Endrias Bridge" -StartupType Automatic |
| Query status | Get-Service "Endrias Bridge" |
stop_service_on_exit).Manual Run
You can trigger initialization manually without touching the service.
Normal run (uses migrationbridge.conf)
migrationbridge run
Debug run (verbose logging to console)
migrationbridge --debug run
Run with a custom config file
migrationbridge --config D:\custom\myconfig.conf run
Print version
migrationbridge version
retry_count = 1 and retry_interval = 1 to migrationbridge.conf. This avoids waiting 45+ seconds when no cloud IMDS is reachable.Configuration Reference
Edit C:\Users\Endrias.bekele\AppData\Local\Programs\Endrias Bridge\migrationbridge.conf. All keys live under [DEFAULT].
| Key | Default | Description |
|---|---|---|
username | Admin | Local Windows account to create |
groups | Administrators | Comma-separated group memberships |
inject_user_password | true | Generate a random password if none in metadata |
config_drive_cdrom | true | Scan CD-ROM drives for ConfigDrive |
config_drive_raw_hhd | true | Scan HDD partitions for ConfigDrive |
http_metadata_url | http://169.254.169.254 | IMDS base URL |
retry_count | 5 | HTTP metadata retry attempts |
retry_interval | 4 | Seconds between retries |
allow_reboot | true | Let plugins trigger a system reboot |
stop_service_on_exit | true | Stop the service after plugins finish |
debug | false | Enable DEBUG-level logging |
log_file | C:\Windows\Temp\migrationbridge.log | Log output path |
first_boot_file | C:\Windows\Temp\migrationbridge_firstboot | Marker file path |
metadata_services | (comma list) | Ordered list of metadata source classes to try |
plugins | (comma list) | Ordered list of plugin classes to run |
db_migration_chunk_size | 100000 (default) | Rows per fetch + INSERT batch for the Endrias Bridge data copy. 100,000 is the confirmed production value (Azure SQL → AWS RDS; actual row widths all < 2,000 bytes). 50,000 is the proven safe floor. Lower to 20,000–25,000 for wide / BLOB / nvarchar(MAX) tables. Takes effect immediately — the app auto-loads migrationbridge.conf at startup. Full sizing guide: Endrias Bridge_Troubleshooting.html → Performance Tuning. |
Plugins Reference
Plugins run in the order listed in migrationbridge.conf. You can add, remove, or reorder them.
Reads hostname from metadata, renames the computer via PowerShell Rename-Computer, and signals a reboot.
Creates a local Windows account and adds it to the configured groups using net user / net localgroup.
Sets the password from metadata. If none is provided and inject_user_password=true, generates a cryptographically random 16-char password.
Configures static IP addresses, gateway routes, and DNS server addresses via PowerShell New-NetIPAddress / Set-DnsClientServerAddress.
Appends public keys from metadata into C:\Users\{username}\.ssh\authorized_keys. Skips keys already present.
Auto-detects script type (PowerShell #ps1, Bash #!/bin/bash, CMD rem cmd), writes to a temp file, and executes it.
Adding a custom plugin
Create a new file in migrationbridge\plugins\, inherit from BasePlugin, and add its dotted path to plugins = in the config:
# migrationbridge/plugins/my_plugin.py from migrationbridge.plugins.base import BasePlugin, PLUGIN_EXECUTED import logging LOG = logging.getLogger(__name__) class MyPlugin(BasePlugin): def execute(self, service, shared_data): LOG.info('Hello from MyPlugin!') # shared_data['username'] has the created user # service.get_meta_data() has raw metadata dict return PLUGIN_EXECUTED
# migrationbridge.conf — append to plugins list plugins = ...existing...,migrationbridge.plugins.my_plugin.MyPlugin
Metadata Sources
Endrias Bridge tries each source in order and uses the first one that loads successfully.
| Source | Class | How it works | Use with |
|---|---|---|---|
| ConfigDrive v2 | configdrive.ConfigDriveSource |
Scans all drive letters for openstack\latest\meta_data.json |
OpenStack, Proxmox, oVirt, Hyper-V |
| OpenStack HTTP | http_source.HttpSource |
GET 169.254.169.254/openstack/latest/meta_data.json |
OpenStack Nova, DevStack |
| EC2 | ec2.EC2Source |
GET 169.254.169.254/latest/meta-data/local-hostname |
AWS EC2, compatible clouds |
BACPAC Seed — When to Use It
BACPAC Seed uses SqlPackage to export a table from the source database into a .bacpac
file (a ZIP containing BCP character-encoded data), then loads it into the target via
bcp.exe. It is designed for tables that cannot be migrated efficiently by the
PK-range path alone.
| Use BACPAC Seed when… | Use main migration PK-range when… |
|---|---|
| Table has no numeric/sequential PK (bcp is the only bulk path) | Table has an identity or sequential integer PK |
| Table rows are too wide (LOB, max-length varchar) for efficient row-by-row copy | Row width is <8 KB average |
| You have a pre-exported .bacpac on disk (Import Only mode saves re-export time) | Target is Azure SQL PaaS / AWS RDS — falls back to PK-range automatically |
BULK INSERT from a local file is not permitted.
The tool auto-detects this and falls back to PK-range copy. BACPAC Seed still works on these
targets — it uses bcp with a direct server connection string rather than a local file path.BACPAC Seed — Correct Order of Operations
Correct sequence:
-
Run Migration (all tables)
Creates all 154+ tables on target in FK-dependency order, copies all rows via PK-range, and verifies constraints. Temporal versioning is left OFF — Re-apply enables it later.
-
Deploy code fix (if not already on build ≥ 72)
Copy updated
db_migration.pyto the running tool path. Enables bcp character mode, file-lock retry, and FK 2714 silencing. -
Free disk space on the BACPAC output drive
Need at least 2× the largest table size as free space. Delete stale .bacpac files from prior runs.
-
Run BACPAC Seed
All target tables exist — bcp can import without NativeError 208. Tables already copied by PK-range are detected and skipped automatically (row-count guard).
-
Run Re-apply Schema Objects
Re-enables SYSTEM_VERSIONING on temporal tables and recompiles all views, functions, procedures, and triggers. Succeeds because all base tables now have data.
-
Validate
Row counts, temporal versioning state, FK spot checks. See validation queries in Troubleshooting Guide.
BACPAC Seed — Step-by-Step
1. Verify main migration completed
-- Run on target; expect all source tables to appear SELECT COUNT(*) FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_TYPE = 'BASE TABLE'; -- Expected: matches source table count (e.g. 154)
2. Check disk space
Get-PSDrive C # or whichever drive holds the BACPAC output dir # Need Free > 2x size of largest table
3. Clear stale .bacpac files
Get-ChildItem "C:\path\to\bacpac\output" -Filter *.bacpac | Remove-Item -Force
4. Deploy code fix (build ≥ 72)
Copy-Item ` "C:\MigrationT\Migrationtool\migrationbridge\gui\db_migration.py" ` "<OneDrive path>\MigrationTool\migrationbridge\gui\db_migration.py" ` -Force # Restart the EndriasBridge GUI after copying
5. Run BACPAC Seed in the GUI
- Open the BACPAC Seed tab.
- Select the tables to seed (typically the large tables with no PK, or tables with LOB data).
- Set the Output directory to a drive with sufficient free space.
- If you have .bacpac files from a previous run on disk, enable Import Only mode to skip re-export.
- Click Run BACPAC Seed.
Monitor the log. Expect each table to either import rows or report "already on target — skipping."
6. Run Re-apply Schema Objects
- Open the Schema Objects tab.
- Click Re-apply Schema Objects.
- All 38 temporal tables should show
system-versioning ON. - All 166 schema objects should apply with 0 failures.
7. Validate
-- Temporal versioning SELECT OBJECT_SCHEMA_NAME(object_id)+'.'+name AS tbl, temporal_type_desc FROM sys.tables WHERE temporal_type = 2 ORDER BY 1; -- Expected: 38 rows, all SYSTEM_VERSIONED_TEMPORAL_TABLE -- Row count spot checks (use your own table names) SELECT COUNT(*) FROM coredata.Configuration; SELECT COUNT(*) FROM coredata.ProviderType; -- FK chain integrity SELECT TOP 1 pt.ConfigurationId FROM coredata.ProviderType pt JOIN coredata.Configuration c ON pt.ConfigurationId = c.ConfigurationId; -- Returns a row → FK intact
BACPAC Error Quick-Reference
| Error message | Code | Root cause | Fix |
|---|---|---|---|
| Invalid object name 'schema.Table' | 208 | BACPAC ran before main migration created the table | Run main migration first |
| SQLState 22001 — String data, right truncation | 22001 | bcp using -n native mode on character-encoded .bacpac data |
Update to build ≥ 72 (switches bcp to -c character mode) |
| File being used by another process | — | Stale .bacpac from crashed/killed prior run | Delete .bacpac; auto-retried in build ≥ 72 |
| FK references invalid table (1767) | 1767 | FK child table created before parent exists on target | Run main migration first; all parents must exist |
| There is already an object named… (2714) | 2714 | FK already exists on target from prior run; false alarm | Not an error — silenced as informational in build ≥ 72 |
| [Errno 28] No space left on device | Errno 28 | Large .bacpac exhausted free disk space | Delete .bacpac, use drive with ≥2× table size free space |
| system-versioning FAILED (208) on Re-apply | 208 | Re-apply ran before main migration created tables | Complete main migration first, then run Re-apply |
| GetCustomEditIdByConfigurationId FAILED (208) | 208 | Function references a table not yet present on target | Complete main migration first; function compiles after data exists |
Detailed troubleshooting for each error: Troubleshooting Guide — BACPAC sections.
Deploy Code Fix — Build 72 Changes
Build 72 (commit 7093d93) contains three fixes that eliminate the most common
recurring BACPAC failures. If the tool was running before this commit, deploy the fix before
your next BACPAC run.
Changes in build 72
| # | Change | Effect |
|---|---|---|
| 1 | bcp format-file generation switched from -n (native/binary) to
-c (character mode) |
Eliminates SQLState 22001 string truncation on SqlPackage .bacpac data |
| 2 | File-lock auto-retry: if SqlPackage fails with "being used by another process", tool deletes stale .bacpac and retries once | Eliminates manual delete-and-retry step when a prior run crashed mid-export |
| 3 | FK recreate: error 2714 "already exists" downgraded from WARNING to informational "FK already exists (skipped)" | Eliminates noisy WARNING log spam on re-runs where FKs are already present |
Deploy command
# Copy from git working tree to running tool location Copy-Item ` "C:\MigrationT\Migrationtool\migrationbridge\gui\db_migration.py" ` "<OneDrive\path\to\MigrationTool>\migrationbridge\gui\db_migration.py" ` -Force # Verify git commit git -C C:\MigrationT\Migrationtool log --oneline -3
db_migration.py.
Python caches module bytecode — a restart ensures the new code is loaded.Connecting to AWS RDS for SQL Server & Azure SQL Database
AWS RDS for SQL Server and Azure SQL Database are PaaS (Platform as a Service) SQL endpoints. They have specific authentication and connectivity requirements that differ from on-premises SQL Server.
Authentication: SQL Authentication Required
Integrated Security=SSPI will fail with an authentication error on both platforms.When configuring a connection to RDS or Azure SQL in the Setup Wizard (Step 5), a blue info banner is displayed automatically below the credentials fields as a reminder of this requirement.
| Platform | Auth method | Connection string requirement |
|---|---|---|
| AWS RDS for SQL Server | SQL Authentication (username + password) | Integrated Security=false (default). Use the RDS master username or a SQL login created post-provision. |
| Azure SQL Database | SQL Authentication or Azure AD | Integrated Security=false for SQL auth. For Azure AD: Authentication=ActiveDirectoryPassword. |
| On-premises SQL Server | SQL Authentication or Windows Authentication | Both supported. Windows auth requires the staging host to be domain-joined. |
Target Database: No Pre-Creation Required (Auto-Create)
Migration Projects (Jobs tab) automatically creates the target database before migration begins. You do not need to pre-provision the database in the RDS console or Azure Portal.
CREATE DATABASE [name] on the target SQL Server instance. If the database already exists the command is silently skipped. Pre-flight log entries confirm: [preflight] Created target database: <name> or [preflight] Target database already exists — skipping CREATE.To disable auto-create (e.g. when migrating into a pre-existing database), uncheck the checkbox in the Map Targets step.
RDS Connection Checklist
-
Security group inbound rule
Add an inbound rule to the RDS instance's security group allowing TCP port 1433 from the staging host's IP address (or security group). Without this rule the connection will time out.
-
Publicly accessible setting (if staging host is external)
If the staging host is outside the VPC, ensure the RDS instance has Publicly accessible: Yes enabled and DNS resolves to the public endpoint. For staging hosts inside the same VPC, use the private endpoint — no public access required.
-
TLS / certificate settings
RDS for SQL Server enforces TLS. Set
Encrypt=yesin the connection. For development/testing, setTrustServerCertificate=yes. For production, install the Amazon Root CA bundle and setTrustServerCertificate=noto validate the certificate chain. -
Use SQL Authentication
Enter the RDS master username (e.g.
admin) or a SQL login created after provisioning. Do not use Windows credentials or leave the username blank. -
Test connection
Click Test Connection on Step 5 of the wizard before running any migration. A successful test confirms firewall, auth, and TLS settings are all correct.
Pre-Flight Connectivity Check
Endrias Bridge automatically runs a 3-step pre-flight check at the start of every migration. If Step 1 or Step 2 fails the migration is aborted immediately — no data is moved. Step 3 (CDC eligibility) is a warning only.
What it checks
| Step | Check | Pass condition | On failure |
|---|---|---|---|
| 1/3 | Source connection | SELECT 1 succeeds on source engine | Migration aborted — fix source connection on Step 5 of wizard |
| 2/3 | Target connection | SELECT 1 succeeds on target engine | Migration aborted — fix target connection on Step 5 of wizard |
| 3/3 | CDC eligibility | CDC enabled on source DB (SQL Server sources only) | Warning logged — migration continues with watermark/full-copy fallback |
Reading the pre-flight output
Pre-flight results appear at the top of the Migration Log panel:
--- Pre-flight check ---
[1/3] Source connection … OK
[2/3] Target connection … OK
[3/3] CDC eligibility … ENABLED
--- Pre-flight passed — starting migration ---
Failure example:
[2/3] Target connection … FAILED: (pyodbc.OperationalError) Login timeout expired
Pre-flight FAILED — fix connection errors above before retrying.
Common pre-flight failures and remediation
| Error pattern | Cause | Fix |
|---|---|---|
| Login timeout expired | Wrong hostname, port blocked, or VPN not connected | Verify hostname/port; check firewall/security group; connect VPN |
| Login failed for user '...' | Wrong username or password | Correct credentials on Step 5; ensure SQL auth is enabled on server |
| No connection configured on Step 5 | Wizard not completed | Complete wizard Step 5 (source + target DB settings) |
| SSL SYSCALL error / certificate verify failed | TLS cert mismatch (common with RDS) | Add TrustServerCertificate=yes to connection string; or install RDS CA cert bundle |
| CDC eligibility … DISABLED | CDC not enabled on source DB | Run EXEC sys.sp_cdc_enable_db on source; or switch migration mode to Full/Incremental |
Global status pill
The header bar status pill reflects the current state throughout the migration lifecycle:
| Pill text | Color | Meaning |
|---|---|---|
| Idle | Grey | No operation running |
| Migrating… | Amber | Migration in progress |
| Pre-flight failed | Red | Pre-flight check failed — migration did not start |
| Migration complete | Green | Migration finished successfully |
| Migration failed | Red | Fatal error during migration |
| Streaming (…) | Green | CDC replication stream active |
| Schema drift detected | Amber | Source schema changed mid-stream |
| Health check running… | Amber | Configuration Health Check in progress |
| Ready to migrate | Green | All health checks passed |
| Health: N failed | Red | One or more health check failures |
Configuration Health Check
Available in the Tools tab → Configuration Health Check → Run Health Check. Runs 8 automated verifications and reports each with pass / warn / fail status and a specific remediation step.
When to run it
- Before starting any production migration — confirms full stack is ready
- After changing connection strings or credentials
- After installing or upgrading ODBC drivers
- When a migration fails with an unexpected error
- During on-boarding of a new DBA or migration engineer
Health check verifications
| # | Check | Pass condition | Remediation on fail |
|---|---|---|---|
| 1 | Source connection | SELECT @@VERSION returns a value | Correct credentials / hostname on Step 5; check firewall |
| 2 | Target connection | SELECT 1 returns without error | Correct target credentials; check VPN/security group; verify DB exists |
| 3 | Source permissions | User is member of db_datareader (SQL Server) | ALTER ROLE db_datareader ADD MEMBER [login] on source DB |
| 4 | Target permissions | CREATE TABLE / DROP TABLE succeed | Grant CREATE TABLE, INSERT, UPDATE, DELETE, ALTER on target DB |
| 5 | CDC eligibility | is_cdc_enabled = 1 on source DB | EXEC sys.sp_cdc_enable_db + SQL Server Agent running; only required for CDC mode |
| 6 | Staging disk space | ≥ 20 GB free on C:\ | Free disk space; redirect BACPAC output dir to a volume with more space |
| 7 | ODBC Driver 17+ | Driver 17 or 18 present in pyodbc.drivers() | Install ODBC Driver 17 for SQL Server from Microsoft Download Center |
| 8 | bcp.exe on PATH | bcp -v exits without FileNotFoundError | Install SQL Server Command-Line Tools; add bcp.exe folder to system PATH. Non-critical — automatic fallback to PK-range copy. |
Interpreting results
Each row in the health check Treeview is colour-coded:
- ✓ OK (green) — check passed
- ⚠ Warning (amber) — check could not complete or non-critical condition found; migration may still work
- ✗ Failed (red) — critical failure; migration is likely to fail until resolved
The summary line below the button reads: 8 checks: 6 passed, 1 warnings, 1 failed. The header status pill also updates to reflect the worst severity found.
In-App Update Checker
Available in the Tools tab → Check for Updates → Check for Updates. Queries the GitHub Releases API for the latest published version and compares it to the running version (v1.2.0).
Procedure
- Open Endrias Bridge → Tools tab.
- Scroll to the Check for Updates card (purple header).
- Click Check for Updates.
- Wait for the result (typically < 3 seconds on a connected machine).
- If a newer version is available, visit the GitHub Releases page to download the installer.
Expected outcomes
| Message | Meaning |
|---|---|
| You are on the latest version (v1.2.0) | No update needed |
| New version available: vX.Y.Z — visit GitHub to download | Download from GitHub Releases; re-run installer over existing install |
| Check failed: <error> | No outbound internet, proxy blocking api.github.com, or rate limit hit. Check network; try again later. |
CDC Replication — What It Is & When to Use It
Change Data Capture (CDC) is a SQL Server feature that reads the transaction log and captures every INSERT, UPDATE, and DELETE in near-real-time. Endrias Bridge uses CDC to keep a target database in sync with a live source database — enabling a minimal-downtime cutover where you migrate data first, then redirect the application once both sides are identical.
Prerequisites for CDC replication
| Requirement | Details | How to verify |
|---|---|---|
| SQL Server source (on-prem or cloud) | SQL Server 2012+, Azure SQL Database, Azure SQL MI, AWS RDS for SQL Server. CDC is not available on SQL Server Express. | Check the source type in Step 5 of the wizard |
| SQL Server Agent running on source | CDC Agent jobs scan the transaction log. Must be running — on AWS RDS this is managed automatically. | SELECT status_desc FROM sys.dm_server_services WHERE servicename LIKE 'SQL Server Agent%' |
db_owner on the source database |
Required to execute sys.sp_cdc_enable_db and sys.sp_cdc_enable_table. |
SELECT IS_MEMBER('db_owner'); — must return 1 |
| Write permissions on target | The Endrias Bridge user on the target needs db_owner or INSERT, UPDATE, DELETE on all replicated tables. |
Run the Test Connection button on Step 5; attempt a manual INSERT on the target |
| Network connectivity: TCP 1433 open both ways | Endrias Bridge must reach both source and target. TLS/SSL required for Azure SQL and AWS RDS. | Use the Test Connection button; check firewall rules / security group inbound rules on port 1433 |
| Sufficient transaction log space on source | CDC holds the log open during the initial copy window. Ensure ≥ 20% free on the source log drive. | EXEC xp_fixeddrives; on source; check sys.dm_os_volume_stats |
| Initial full migration completed | CDC Stream only applies delta changes — the target must already have a full copy of the data before the stream starts. | Run the Full migration first (see Initial Load Procedure below) |
CDC vs other sync modes
| Mode | How it works | Best for | Limitations |
|---|---|---|---|
| Full | Truncate target, copy all rows | First-time migration, small databases | Downtime = copy time; no incremental |
| Incremental | Watermark column — copies rows newer than last run | Recurring delta loads; any engine | No delete capture; requires timestamp/rowversion column |
| CDC (single-pass) | Reads change tables once, applies net changes | Scheduled job to catch up on deltas | Not continuous; SQL Server source only |
| CDC Stream | LSN tail loop — polls every 250 ms, applies changes as they arrive | Low-downtime cutover; real-time shadow copy | SQL Server source only; CDC must be enabled per-table |
| Scheduled Sync (Replication tab) | Watermark sync on a configurable timer | Any-engine ongoing sync; no CDC required | No delete capture; sync latency = interval setting |
CDC Initial Load Procedure
The initial load is the bulk copy that seeds the target database before the CDC stream starts. The order of operations is critical — you must enable CDC on the source before starting the full copy, so that no changes are lost during the copy window. If you enable CDC after the copy, any rows written to the source while the copy was running will be missing from the target permanently.
-
Step 1 — Enable CDC at the database level
Run this once on the source database. Requires
db_ownerorsysadmin.-- Run on SOURCE database USE [YourSourceDB]; EXEC sys.sp_cdc_enable_db; -- Verify SELECT name, is_cdc_enabled FROM sys.databases WHERE database_id = DB_ID(); -- Expected: is_cdc_enabled = 1
-
Step 2 — Enable CDC on each table you want to track
Repeat for every table. You do not need to enable CDC on every table in the database — only the ones you will replicate.
-- Enable CDC on one table EXEC sys.sp_cdc_enable_table @source_schema = N'dbo', @source_name = N'YourTable', @role_name = NULL; -- NULL = no role gate; change if access control needed -- Verify all tracked tables SELECT s.name AS [schema], t.name AS [table], t.is_tracked_by_cdc FROM sys.tables t JOIN sys.schemas s ON s.schema_id = t.schema_id WHERE t.is_tracked_by_cdc = 1 ORDER BY s.name, t.name;ℹ️SQL Server Agent must be running — CDC uses Agent jobs to scan the log. On AWS RDS for SQL Server the Agent is managed automatically. -
Step 3 — Note the current LSN (bookmark)
Record the maximum LSN at the moment CDC was enabled. This is the starting point the stream will use after the initial copy completes.
-- Run immediately after enabling CDC, before starting the full copy SELECT sys.fn_cdc_get_max_lsn() AS start_lsn; -- Save this value — it is your recovery bookmark if anything goes wrong
-
Step 4 — Run the initial full copy in Endrias Bridge
In the Migration tab, set sync mode to Full and click Run Migration. This copies all rows from source to target. CDC is accumulating changes in the background during this copy — that is exactly what you want.
- For large tables (> 100 M rows), enable BCP acceleration and Disable indexes during copy in the Migration options
- For Azure SQL / AWS RDS targets, use PK-range copy (automatic fallback)
- Do not stop CDC or disconnect the source during this step
-
Step 5 — Validate the initial copy
Run these checks before starting the CDC stream:
-- Row count comparison (run on both source and target for each table) SELECT t.name AS table_name, p.rows AS row_count FROM sys.tables t JOIN sys.partitions p ON p.object_id = t.object_id AND p.index_id IN (0,1) ORDER BY p.rows DESC; -- Spot-check: max PK on source vs target for a key table SELECT MAX(Id) AS max_id FROM dbo.YourTable; -- run on both sidesRow counts do not need to match exactly — changes that arrived during the copy will be caught by the CDC stream in the next step.
-
Step 6 — Apply schema objects
If the initial copy did not apply views, stored procedures, functions, or triggers, run Re-apply Schema Objects from the Schema Objects tab now — before starting the CDC stream.
Running the CDC Stream
The CDC stream catches up from the LSN recorded during the initial copy, then continues applying changes in near-real-time (< 1 second latency under normal load).
-
Step 1 — Switch sync mode to CDC Stream
In the Migration tab, set the sync mode radio button to CDC Stream (real-time, SQL Server).
-
Step 2 — Click Run Migration
The tool will:
- Verify CDC is enabled at the database level
- Discover which tables have CDC capture instances
- Skip tables with no capture instance and log them (add them with
sp_cdc_enable_tableif needed) - Start the LSN tail loop — polling every 250 ms
-
Step 3 — Monitor the log
Expected log output while the stream is running:
--- CDC Stream started: 12 table(s) — click "Stop Stream" to end --- [stream] cycle 1: +412 change(s) (total 412) rate=206/s LSN=0x000000001A4F... elapsed 00:00:02 [stream] cycle 2: +38 change(s) (total 450) rate=19/s LSN=0x000000001A51... elapsed 00:00:03 [stream] cycle 3: +0 change(s) -- (idle, no log shown when LSN unchanged) [stream] cycle 7: +5 change(s) (total 455) rate=2/s LSN=0x000000001A52... elapsed 00:00:10
The first few cycles will catch up the accumulated changes from the initial copy window. Once the rate drops to match normal application traffic, the target is live.
-
Step 4 — Monitor the Replication tab (optional)
The Replication tab shows a per-table dashboard with rows synced, current LSN, change rate, and schema drift alerts. You can start the CDC stream from there instead of the Migration tab if you want the live dashboard visible.
What to watch for during the stream
| Signal | Meaning | Action |
|---|---|---|
| Rate > 500 changes/s sustained | Target may fall behind; disk/IOPS limit approaching | Increase target IOPS; reduce source write load temporarily |
| SCHEMA DRIFT on TableName | A column was added or removed on the source while streaming | Stop stream → update target schema → re-run Re-apply Schema Objects → restart stream |
| source read error — retrying | Transient network blip to source | Auto-retries with backoff (5 → 120 s). No action unless it persists > 5 min. |
| Table skipped: no CDC capture instance | CDC not enabled on that specific table | EXEC sys.sp_cdc_enable_table for that table, then restart stream |
Final Cutover Procedure
Cutover is the moment you redirect the application from the source to the target database. The goal is to do this with the shortest possible downtime — typically under 60 seconds. Follow this checklist precisely.
-
Step 1 — Confirm stream is caught up
Wait until the CDC stream log shows a steady low rate (matching normal traffic). The backlog from the initial copy should be fully drained — typical sign is < 10 changes/cycle.
-- Confirm CDC is not lagging: compare max LSN on source vs last checkpoint -- Run on SOURCE: SELECT sys.fn_cdc_get_max_lsn() AS current_source_lsn; -- Run on SOURCE to see CDC Agent job latency: SELECT job_name, last_run_duration, last_run_outcome_desc FROM msdb.dbo.sysjobhistory sjh JOIN msdb.dbo.sysjobs sj ON sj.job_id = sjh.job_id WHERE sj.name LIKE 'cdc%' ORDER BY sjh.run_date DESC, sjh.run_time DESC;
-
Step 2 — Stop application writes to the source (maintenance window begins)
This is the start of downtime. Options:
- Take the application offline / put it in maintenance mode
- Revoke write permissions on the source database:
-- Prevent new writes on source (run on SOURCE) REVOKE INSERT, UPDATE, DELETE ON SCHEMA::dbo TO [AppUser]; -- Remember to re-grant if cutover is rolled back
- Rename the source database (forces connection failures, immediate)
-
Step 3 — Wait for the stream to drain
Watch the Migration log or Replication tab. Wait until you see two or three consecutive idle cycles (no new changes logged). This means all in-flight transactions have been replicated to the target.
[stream] cycle 91: +3 change(s) (total 1,203,882) rate=0/s LSN=0x0000000021A1... [stream] cycle 92: +1 change(s) (total 1,203,883) rate=0/s LSN=0x0000000021A2... [stream] cycle 93: (no new LSN — source is quiet) [stream] cycle 94: (no new LSN — source is quiet) -- ✅ Two idle cycles: source is drained. Safe to stop.
-
Step 4 — Validate row counts on target
Run these queries on both source and target. They must match (or target should be within 0 rows, since writes are stopped).
-- Run on BOTH source and target for each critical table: SELECT t.name AS table_name, SUM(p.rows) AS row_count FROM sys.tables t JOIN sys.partitions p ON p.object_id = t.object_id AND p.index_id IN (0, 1) GROUP BY t.name ORDER BY SUM(p.rows) DESC; -- Spot-check: max and min PK on a key table (run on BOTH sides) SELECT MIN(Id) AS min_id, MAX(Id) AS max_id, COUNT(*) AS cnt FROM dbo.YourMostImportantTable;⚠️If row counts are off by more than a few rows, do NOT proceed. Resume the stream, investigate the discrepancy, and retry drain. -
Step 5 — Stop the CDC stream
Click Stop Stream in the Migration tab or Replication tab. Wait for the log confirmation:
--- CDC Stream stopped: 1,203,883 change(s) in 02:14:07 (149.1 changes/s) ---
-
Step 6 — Redirect the application to the target
Update the application connection string (or DNS alias / CNAME) to point to the target database. Smoke-test core functionality before opening traffic.
- Verify logins exist on the target: run the Users & Logins tab comparison
- Verify application can write: run a test transaction end-to-end
- Check for errors in the application log for the first 5 minutes
-
Step 7 — Post-cutover validation
-- Verify no replication errors remain on target -- Check for orphaned rows in FK-parent tables: SELECT t.name AS table_name, p.rows AS row_count FROM sys.tables t JOIN sys.partitions p ON p.object_id = t.object_id AND p.index_id IN (0,1) ORDER BY p.rows DESC; -- Verify temporal tables have SYSTEM_VERSIONING ON (SQL Server targets) SELECT name, temporal_type_desc FROM sys.tables WHERE temporal_type IN (2) -- 2 = SYSTEM_VERSIONED_TEMPORAL_TABLE ORDER BY name; -- Verify schema objects applied SELECT type_desc, COUNT(*) AS cnt FROM sys.objects WHERE is_ms_shipped = 0 GROUP BY type_desc ORDER BY type_desc;
Rollback plan
| Scenario | Action |
|---|---|
| Row count mismatch found before redirect | Restore write access on source (GRANT back to AppUser) → resume stream → re-drain |
| Application errors after redirect (first 30 min) | Point app back at source connection string → investigate target issue → re-attempt cutover |
| Critical data missing on target post-cutover | Use source as read reference; run missing-row query; INSERT specific rows from source to target manually |
EXEC sys.sp_cdc_disable_db;
Replication Tab Guide
The Replication tab is a dedicated control panel for ongoing replication — separate from the Migration tab. Use it when you want a persistent, always-visible dashboard for a live CDC stream or a recurring scheduled sync.
CDC Stream panel
| Control | What it does |
|---|---|
| Poll interval (ms) | How often to check sys.fn_cdc_get_max_lsn(). Default 250 ms. Lower = faster reaction; higher = lower idle CPU. 100–500 ms is the practical range. |
| Start CDC Stream | Connects to source and target, reflects metadata, discovers CDC-enabled tables, starts the LSN tail loop in a background thread. |
| Stop Stream | Sets the stop event; the current cycle drains cleanly before the thread exits. Final change count is logged. |
| LSN display | Shows the last processed Log Sequence Number in hex. Advances with every cycle that has changes. |
| Rate display | Rolling 60-second changes/second. Turns red above 500/s. |
| Schema drift alert | Red banner appears if a tracked table's column list changes during the stream. The table is suspended until you stop, fix, and restart. |
Scheduled Sync panel
| Control | What it does |
|---|---|
| Sync every N minutes | How often to run a watermark sync pass. Minimum 1 minute. For near-real-time on non-SQL-Server sources, set to 1. |
| Start Scheduled Sync | Discovers all tables present on both source and target, then runs sync_table_smart() per table on the configured interval. |
| Stop | Sets the stop event; the current cycle completes before the thread exits. |
| Next run / Last run | Countdown timer and timestamp of most recent completed cycle. |
Per-table status dashboard
The right-hand Treeview updates after every cycle:
| Column | Meaning |
|---|---|
| Table | Table name (schema-qualified if applicable) |
| Mode | cdc — native SQL Server CDC used; watermark — timestamp/rowversion column used; no_sync — no watermark found, table skipped |
| Rows Synced | Cumulative rows inserted or updated since stream started (this session) |
| Last Sync | Wall-clock time of last successful change application for this table |
Common Issues
Service fails to start
-
Check the Windows Event Log
Run:
Get-EventLog -LogName Application -Source "Endrias Bridge" -Newest 10 -
Verify pywin32 is installed
Run:
python -c "import win32service; print('OK')" -
Run post-install step for pywin32
Run as Admin:
python Scripts\pywin32_postinstall.py -install
Plugins failing with "access denied"
Metadata not being found (HTTP timeout)
-
Reduce retry count during testing
Set
retry_count = 1andretry_interval = 1inmigrationbridge.confto avoid long waits. -
Test IMDS reachability
Run:
Test-NetConnection -ComputerName 169.254.169.254 -Port 80 -
Use ConfigDrive instead
Mount a ConfigDrive ISO to the VM. Endrias Bridge will detect it automatically before trying HTTP.
Hostname not changing
The hostname change requires a reboot to take effect. If allow_reboot = false in the config, the rename is queued but the machine won't restart automatically. Set allow_reboot = true or reboot manually.
Re-run everything from scratch
# Remove first-boot marker (as Administrator) Remove-Item C:\Windows\Temp\migrationbridge_firstboot -Force Remove-Item C:\Windows\Temp\migrationbridge.log -Force Restart-Service "Endrias Bridge"
Windows Event Log
When running as a service, Endrias Bridge writes to the Application event log.
View Endrias Bridge events
Get-EventLog -LogName Application -Source "Endrias Bridge" -Newest 20
View all recent Application errors
Get-EventLog -LogName Application -EntryType Error -Newest 20
Open Event Viewer GUI
eventvwr.msc # Navigate to: Windows Logs → Application → filter by Source = Endrias Bridge
debug = true in migrationbridge.conf and check migrationbridge.log for the most detailed output. The Event Log captures service start/stop events; the log file captures everything else.CT Sync — What It Is & When to Use It
CT Sync uses SQL Server Change Tracking to replicate only the rows that changed since the last sync — without reading the transaction log (no CDC license or SQL Server Agent required). It is lighter and simpler than CDC Stream.
Use CT Sync when
- You need continuous or scheduled delta sync from Azure SQL → AWS RDS
- CDC is not available or not licensed on the source
- You want near-real-time replication with < 1 second latency (tail-loop mode)
- You do not need the full before/after row image (CT only records which PKs changed)
CT Sync vs CDC Stream
| Feature | CT Sync | CDC Stream |
|---|---|---|
| Source requirement | Change Tracking enabled on DB + table | CDC enabled on DB + table, SQL Server Agent running |
| Reads transaction log | No | Yes |
| Before-image (old value) | No — PK + operation only | Yes — full row before/after |
| Latency | < 1 s (tail-loop) / scheduled | < 1 s |
| Retention window risk | Yes — CT version can expire | Yes — LSN log can truncate |
Step 1 — Enable Change Tracking on Source
Run on the source Azure SQL database. Requires ALTER DATABASE permission.
Enable at database level
-- Replace YourDatabase with the source DB name
ALTER DATABASE [YourDatabase]
SET CHANGE_TRACKING = ON
(CHANGE_RETENTION = 3 DAYS, AUTO_CLEANUP = ON);
Enable on each table to sync
-- Repeat for every table included in CT Sync
ALTER TABLE [dbo].[YourTable]
ENABLE CHANGE_TRACKING
WITH (TRACK_COLUMNS_UPDATED = OFF);
Verify CT is enabled
-- Check database-level CT SELECT name, is_change_tracking_on FROM sys.databases WHERE name = DB_NAME(); -- Check per-table CT SELECT OBJECT_NAME(object_id) AS table_name, is_track_columns_updated_on FROM sys.change_tracking_tables;
CHANGE_RETENTION to a minimum of 30 DAYS. This covers any operational pause (weekends, incidents, freeze windows) without requiring a reseed. Example: ALTER DATABASE … SET CHANGE_TRACKING = ON (CHANGE_RETENTION = 30 DAYS, AUTO_CLEANUP = ON). See Live Migration Runbook — Retention Quick-Reference for all-engine commands.Step 2 — Running CT Sync (GUI)
- Open Endrias Bridge → Migration tab.
- Connect source (Azure SQL) and target (AWS RDS) — Step 5 connection fields.
- Select the tables to sync (same tables that have CT enabled on source).
- Set Sync Mode to CT Sync (Change Tracking).
- Ensure Migrate Data is ticked; schema migration is not required on subsequent runs.
- Click Run Migration.
First run behavior
If no CT version checkpoint exists for a table, Bridge saves the current CHANGE_TRACKING_CURRENT_VERSION() as the baseline. No data is copied on the first run — subsequent runs will pick up all changes since that baseline.
To seed data before CT Sync, run a Full migration first, then switch to CT Sync mode.
Expected log output
CT Sync: dbo.Claims — 142 changes (87 ins, 43 upd, 12 del) applied CT Sync: dbo.Members — 0 changes since version 88421 CT Sync: dbo.Providers — 8 changes (8 ins, 0 upd, 0 del) applied
Step 3 — Continuous Tail-Loop Mode
CT Sync tail-loop polls for new changes every second and applies them immediately — latency < 1 second for active tables.
Prerequisites
- CT enabled on all selected tables (Step 1)
- Full migration completed to seed initial data
- Source database actively receiving writes
Starting the loop
- Set sync mode to CT Sync (Change Tracking).
- Click Run Migration. The loop starts and runs continuously.
- Monitor the log — each cycle logs the number of changes applied per table.
- Click Stop to cleanly terminate the loop. The current version is saved to the checkpoint file.
CLI Headless Mode SOP
Run migrations without a GUI — usable in Azure DevOps, PowerShell scripts, Windows Task Scheduler, and CI/CD pipelines.
Basic syntax
MigrationBridge migrate --src-type azuresql --src-host yourserver.database.windows.net --src-port 1433 --src-db SourceDB --src-user migrator --src-pass $(SRC_PASS) --tgt-type rds --tgt-host rds-instance.xxxx.rds.amazonaws.com --tgt-port 1433 --tgt-db TargetDB --tgt-user migrator --tgt-pass $(TGT_PASS) --mode full --schema --data --verify --report migration_report.html
Key flags
| Flag | Values | Description |
|---|---|---|
--mode | full, incremental, ct_sync, cdc_stream | Sync mode |
--schema | (flag) | Migrate schema objects |
--data | (flag) | Migrate table data |
--verify | (flag) | Run post-migration row count + checksum verification |
--report PATH | .html or .txt | Write migration + verify report to file |
--tables T1 T2 | space-separated | Limit to specific tables; default = all |
Exit codes
| Code | Meaning |
|---|---|
0 | All tables migrated (and verified, if --verify passed) successfully |
1 | One or more errors — check log or report file for details |
Azure DevOps pipeline task
# azure-pipelines.yml
- task: PowerShell@2
displayName: 'DB Migration: Azure SQL → AWS RDS'
inputs:
script: |
MigrationBridge migrate \
--src-type azuresql --src-host $(AZURE_HOST) --src-db $(SRC_DB) \
--src-user $(SRC_USER) --src-pass $(SRC_PASS) \
--tgt-type rds --tgt-host $(RDS_HOST) --tgt-db $(TGT_DB) \
--tgt-user $(TGT_USER) --tgt-pass $(TGT_PASS) \
--mode full --schema --data --verify --report migration_report.html
if ($LASTEXITCODE -ne 0) { Write-Error "Migration failed"; exit 1 }
Post-Migration Verification SOP
Verification compares row counts and MD5 checksums between source and target after every migration. It runs automatically when Verify after migration is checked in the GUI, or when --verify is passed in CLI mode.
What is checked
- Row count — source vs target row count per table
- MD5 checksum — hash of all row content per table; detects silent data corruption
Enabling in GUI
- Migration tab → tick Verify after migration (row counts + checksums).
- Run migration. Verification runs automatically after data copy completes.
- Results appear in the migration log and in the HTML report (if report output is configured).
Reading the report
Verification Results ✓ dbo.Claims rows=14,220,884 checksum=OK ✓ dbo.Members rows=3,441,002 checksum=OK ✗ dbo.Orders rows source=5,002,441 target=5,002,438 MISMATCH
Any table showing MISMATCH must be investigated. Common causes: writes to source during migration, type conversion differences (checksum only), or a network error that interrupted one batch.
Validation queries after verification
-- On target RDS: confirm row count for a flagged table SELECT COUNT(*) FROM [dbo].[Orders]; -- On source Azure SQL: same SELECT COUNT(*) FROM [dbo].[Orders]; -- If counts differ: find orphaned PKs SELECT id FROM [dbo].[Orders] -- on source EXCEPT SELECT id FROM [dbo].[Orders]; -- on target (run via linked server or re-migrate)