WordPress site migration · streaming · resumable

How Reprint works

Reprint clones an entire WordPress site — files, database, and server configuration — over plain HTTP, in resumable streaming chunks designed to survive the tightest shared-hosting limits. This document maps every block of the repo and the patterns that hold them together.

Producer ↔ consumer over HTTP Cursor-based resumption multipart/mixed streaming HMAC-SHA256 auth URL rewriting on import MySQL or SQLite targets

01What Reprint is

Reprint copies a live WordPress site from a source (a remote site you don't fully control — often locked-down shared hosting) to a target (your laptop, or another host). The whole transfer happens over ordinary HTTP requests against a single WordPress plugin endpoint, so it works even where SSH, direct DB access, and raw PHP execution are blocked.

🖨️

One command

reprint.phar pull https://example.com pulls files + DB + config and starts a local copy you can open in a browser.

⏸️

Resumable everywhere

Every transfer is broken into chunks with a JSON cursor. Interrupt it, re-run the same command, it continues from the last byte.

🪶

Memory & time aware

The server stops a request before hitting the host's memory/time limit and hands back a cursor — never an OOM kill.

The architecture is a textbook producer–consumer split with a streaming pipe between them:

RoleLives inWhat it does
Producer (server)packages/reprint-exporter, shipped as the reprint-exporter-wp WordPress pluginAnswers authenticated HTTP requests by streaming SQL dump fragments and file contents, pausing on a budget and emitting a resume cursor.
Consumer (client)packages/reprint-importer, shipped as reprint.pharA CLI that drives the migration: indexes, diffs, downloads, rewrites URLs, applies the DB, and generates runtime config.
The pipeHTTPmultipart/mixed, gzip-compressed, HMAC-signed, with cursors carried in headers.

02System overview

At the highest level there are two PHP processes that never share a filesystem or a database — they only exchange HTTP requests. The source runs inside WordPress; the target is a standalone CLI.

flowchart LR
  subgraph SRC["🖥️  MIGRATION SOURCE — remote WordPress site"]
    direction TB
    WP["WordPress front controller
?reprint-api"]:::wp PLG["reprint-exporter-wp
plugin shell · HMAC auth"]:::plg HTTPS["Site_Export_HTTP_Server
router + ResourceBudget"]:::core MDP["MySQLDumpProducer"]:::prod FTP["FileTreeProducer"]:::prod DB[("MySQL / SQLite")]:::store FS[("Filesystem")]:::store WP --> PLG --> HTTPS HTTPS --> MDP --> DB HTTPS --> FTP --> FS end subgraph TGT["💻 MIGRATION TARGET — your machine / a host"] direction TB CLI["reprint.phar — CLI dispatch"]:::cli IC["ImportClient
(state machine, ~9k LOC)"]:::core MSP["MultipartStreamParser"]:::prod REW["URL Rewriter +
Query Stream"]:::prod HOST["Host analyzer →
RuntimeManifest → applier"]:::prod OUT[("--fs-root files
db.sql / target DB
runtime.php")]:::store CLI --> IC --> MSP IC --> REW --> OUT IC --> HOST --> OUT end HTTPS == "multipart/mixed · gzip · X-Cursor
HMAC-SHA256 signed" ==> MSP IC -. "GET/POST ?endpoint=… + X-Export-Cursor" .-> HTTPS classDef wp fill:#eef4ff,stroke:#9bb8f0,color:#1c3a6e; classDef plg fill:#f3effe,stroke:#c0adfb,color:#46329b; classDef core fill:#0c1322,stroke:#0c1322,color:#fff; classDef prod fill:#eafaf8,stroke:#7fd8cf,color:#0a6b62; classDef cli fill:#fff4e6,stroke:#f0c182,color:#8a5410; classDef store fill:#f1f4fa,stroke:#cdd6e6,color:#37445c;
The source never initiates a connection. The target pulls everything, one resumable request at a time.
The central insight A single request can never assume it will finish. Both producers are written so that any request can be cut short by a budget, a crash, or a network blip — and the next request picks up exactly where the last one stopped. Everything else in the codebase is downstream of that one constraint.

03Repository map

The repo is a monorepo: two Composer packages (exporter + importer), a WordPress plugin that bundles the exporter, a Playground-based web UI, and the build tooling that produces the release artifacts.

reprint/
├── packages/
│   ├── reprint-exporter/src/          # PRODUCER (server)
│   │   ├── export.php                 # endpoint functions + streaming engine (~3.8k LOC)
│   │   ├── class-http-server.php      # Site_Export_HTTP_Server — request router
│   │   ├── class-mysql-dump-producer.php   # SQL dump state machine (~1.5k LOC)
│   │   ├── class-file-tree-producer.php    # file content streamer
│   │   ├── class-hmac-server.php / -client.php   # auth
│   │   └── class-sqlite-driver-pdo.php / -wpdb-driver-pdo.php / -pdo-polyfill.php  # DB fallbacks
│   │
│   └── reprint-importer/src/          # CONSUMER (client)
│       ├── import.php                 # the whole CLI engine (~12k LOC)
│       └── lib/
│           ├── pull/class-pull.php          # the high-level pull orchestrator
│           ├── url-rewrite/                 # the db-apply URL rewriting pipeline
│           ├── host/                        # source-host analyzers + RuntimeManifest
│           ├── target-runtime/              # runtime appliers (nginx / php-builtin / playground)
│           ├── mysql-query-stream/          # statement splitters (naive + fast)
│           └── external-merge-sort.php      # on-disk sort for huge index files
│
├── reprint-exporter-wp/               # WordPress plugin distribution (index.php + lib.php + admin UI)
├── reprint-ui/                        # WP.com → Playground browser wizard
├── lib/sqlite-database-integration/   # git submodule: WordPress' MySQL→SQLite driver
├── importer/import.php                # thin wrapper to run the importer from source
├── bin/                               # build-reprint-phar.sh, phar-stub.php, stamp-plugin-version.sh
└── box.json                           # Box config — compiles reprint.phar
Scale note import.php is a single ~12,000-line file built almost entirely around one class, ImportClient. That's deliberate — the phar must run as a self-contained script on hosts where you can't composer install. The supporting lib/ subsystems are pulled in by the build.

04Export side — the producer (server)

The export engine lives in packages/reprint-exporter and runs inside WordPress via the reprint-exporter-wp plugin. Its job: authenticate a request, pick the right producer, and stream a chunked response that can pause and resume.

Why a WordPress plugin, and why ?reprint-api

Many shared hosts (SiteGround, GoDaddy, WP Cloud) block direct PHP execution inside wp-content/plugins/ at the web-server level — a 403 before PHP ever runs. So the plugin doesn't expose its own URL. Instead it routes through WordPress's own front controller, which hosts never block:

  • reprint-exporter-wp/index.php checks $_GET['reprint-api'] (canonical) or the legacy alias $_GET['site-export-api'] at plugin-load time — the earliest interception point a normal plugin gets — and calls _site_export_handle_api_request() then exit.
  • If it's not an API request, it loads the admin settings UI instead (wordpress/site-export.php).

The request lifecycle

sequenceDiagram
  autonumber
  participant C as ImportClient (target)
  participant WP as WordPress + plugin
  participant H as Site_Export_HTTP_Server
  participant A as Site_Export_HMAC_Server
  participant E as endpoint_* (export.php)
  participant P as Producer
  C->>WP: GET /?reprint-api&endpoint=sql_chunk
X-Auth-* headers, X-Export-Cursor WP->>H: handle_cors_headers_and_terminate_on_options('*') Note over H: OPTIONS short-circuits
(preflight has no creds) WP->>A: _site_export_verify_hmac(secret) A-->>WP: timestamp ±300s? nonce ok?
HMAC + content-hash match? Note over A: fail → 403/503 JSON, exit WP->>H: serve(['default_directory'=>ABSPATH]) H->>H: parse_http_config() · normalize_config()
base64-decode X-Export-Cursor → JSON H->>H: build ResourceBudget (time + memory) H->>E: dispatch(endpoint, config, budget) E->>P: new Producer(pdo/paths, json_cursor) loop until producer finished OR budget exhausted P-->>E: next fragment / chunk E-->>C: multipart part — X-Chunk-Type, X-Cursor, gzip end E-->>C: completion part — X-Status: complete | partial
Auth happens out-of-band (HMAC), before any producer is touched. CORS is wildcard-open precisely because the real gate is the signature, not the origin.

The five endpoints

The client selects one with the endpoint parameter. They're registered in Site_Export_HTTP_Server::default_handlers():

EndpointHandlerWhat it streams
preflightendpoint_preflightPlain JSON (not multipart): WP version, PHP/MySQL versions, memory limits, FS access, DB connectivity, plugins, themes, directory layout, multisite. Every other command refuses to run without it.
db_indexendpoint_db_indexTable metadata (name, estimated rows, data/index bytes, engine, collation) from INFORMATION_SCHEMA, paginated by TABLE_NAME > :last.
sql_chunkendpoint_sql_chunkSQL dump fragments produced by MySQLDumpProducer, gzip multipart.
file_indexendpoint_file_indexA depth-first directory listing in JSON batches: {path, ctime, size, type, target}. Carries its own stack-based cursor.
file_fetchendpoint_file_fetchThe bytes of a client-supplied list of paths, via FileTreeProducer.
Server-side directory dedup endpoint_file_index guards against the pathological symlink layouts of WP Cloud Atomic — e.g. /srv/htdocs/srv → /srv (an infinite cycle) or /wordpress/ and /srv/htdocs/wordpress/ resolving to the same place. should_skip_index_root() compares each directory's realpath() against the scheduled roots and skips duplicates/parents before traversal recurses into them.

Streaming + database access

GzipOutputStream

Output buffering, zlib.output_compression, and proxy buffering (X-Accel-Buffering: no) are all disabled so a custom incremental gzip stream controls flushing: deflate_add(ZLIB_NO_FLUSH) while writing a part, ZLIB_SYNC_FLUSH at part boundaries. This keeps memory flat regardless of payload size.

Errors become chunks, not 500s

Once the gzip stream is open you can't send an HTTP error code. A global $streaming_context lets the error / exception / shutdown handlers inject an X-Chunk-Type: error part mid-stream instead. The completion chunk is itself wrapped in try/catch so a broken stream is logged, never fatal.

The PDO fallback ladder

The producers only ever call PDO-shaped methods (query, prepare, quote, fetch, bindValue, execute). create_db_connection() hands them one of three objects with an identical surface, so the producer code never branches:

flowchart TD
  Q{What does the host have?}:::q
  Q -->|"SQLite site
(SQLITE_DB_DROPIN_VERSION)"| S["SqliteDriverPDO
wraps WP_SQLite_Driver"]:::a Q -->|"no ext-pdo_mysql"| W["WpdbDriverPDO
wraps global \$wpdb"]:::a Q -->|"ext-pdo_mysql present"| R["real PDO
mysql DSN, utf8mb4"]:::a S --> M["MySQLDumpProducer
(same code path either way)"]:::m W --> M R --> M PP["class-pdo-polyfill.php
defines PDO / PDOStatement / PDOException
so PDO:: constants don't fatal on PDO-less hosts"]:::p PP -.underpins.-> S PP -.underpins.-> W classDef q fill:#fff4e6,stroke:#f0c182,color:#8a5410; classDef a fill:#eafaf8,stroke:#7fd8cf,color:#0a6b62; classDef m fill:#0c1322,stroke:#0c1322,color:#fff; classDef p fill:#f3effe,stroke:#c0adfb,color:#46329b;
A WordPress site can be exported even on a host with no PDO MySQL extension at all — the wpdb adapter bridges the gap.

05Import side — the consumer (client)

The importer is a single CLI (reprint.phar) built around the ImportClient class. It has two layers: a set of low-level commands (one network concern each), and a high-level pull orchestrator that chains them.

Command dispatch

The bottom of import.php parses argv, applies legacy aliases (files-sync→files-pull, db-sync→db-pull, …), constructs new ImportClient($url, $stateDir, $fsRoot), and calls run($options), which routes to a run_<command>() method. EXIT_AFTER_IMPORT gates whether the process exits or returns (the latter for the Playground embedder).

CommandLevelRole
pullhighThe full resumable clone — chains everything below into one pipeline.
install-exporterhighPrints the plugin download URL + install steps. No network.
preflight / preflight-assertlowProbe the remote, cache the result, print JSON / a human pass-fail.
files-indexlowStream the remote directory tree to .import-remote-index.jsonl. No file contents.
files-pulllowIndex → diff → download changed files into --fs-root. Delta on re-run.
db-indexlowTable metadata → db-tables.jsonl.
db-pulllowStream the SQL dump to db.sql, stdout, or a live MySQL connection.
db-applylowExecute db.sql into MySQL/SQLite, rewriting URLs in one pass.
db-domainslowList domains discovered in the dump (for choosing --rewrite-url maps).
flat-docrootlowReassemble a non-standard layout into a normal WP docroot via symlinks.
apply-runtimelowGenerate server config (runtime.php + nginx.conf / start.sh) from preflight data.

Exit codes are the control flow

Because any request can be cut short, the low-level commands don't loop internally. They lean on three exit codes and let the caller (a shell loop, or the pull orchestrator) re-invoke:

✅

0 — complete

state.status === 'complete'. The stage is done.

🔁

2 — partial

Progress was made, call me again. Cursor is saved; the next run resumes.

❌

1 — fatal

An exception bubbled to the bootstrap; a JSON error is written to stderr.

The streaming HTTP client

One primitive does all the networking: ImportClient::fetch_streaming(). It's built on cURL with deliberate choices for hostile networks:

  • No total timeout — a stream can legitimately run 20+ minutes. Instead CURLOPT_LOW_SPEED_LIMIT=1 / LOW_SPEED_TIME=300 detect a genuinely stalled connection.
  • Cursor redundancy — the cursor goes out both as the X-Export-Cursor header and a cursor query param, because some proxies strip custom headers.
  • Browser-mimicking headers — a real-looking User-Agent and Sec-Fetch-* headers, because some WAFs block requests that carry custom auth headers under a non-browser UA.
  • Actionable diagnostics — diagnose_http_error() turns redirects, WAF HTML pages, and JSON error payloads into human messages, and distinguishes a retryable timeout from a hard failure.

MultipartStreamParser — incremental, never buffered

cURL's WRITEFUNCTION feeds bytes into MultipartStreamParser as they arrive. It's a three-state machine — BOUNDARY → HEADERS → BODY — that emits two events: streaming body bytes (large files go straight to disk) and a complete event at each part boundary. It keeps a small tail when a boundary might be split across chunks, and a 64 MB buffer cap catches malformed responses (a WAF page with no boundary).


06The wire protocol

Two design decisions define the protocol: how requests are authenticated, and how responses are framed so they can be split, paused, and resumed.

HMAC authentication

There are no cookies or sessions. Every request is signed with a shared secret using a scheme that survives cURL's unpredictable multipart framing:

signature = HMAC-SHA256( nonce + timestamp + SHA256(body) , secret )   // hex

The signature covers a hash of the body (sent as X-Auth-Content-Hash), not the raw bytes — so for file uploads the client signs the logical content the server can independently reconstruct, regardless of how cURL frames the multipart request. Site_Export_HMAC_Server::verify_globals() checks all four X-Auth-* headers are present, the timestamp is within ±300s (replay window), the nonce is ≥16 chars, and runs every comparison through hash_equals() (constant-time). The secret comes from secret.php (file override, takes precedence) or the site_export_secret WordPress option (set via the admin UI / REST).

multipart/mixed framing

Responses are Content-Type: multipart/mixed; boundary="boundary-<32 hex>", usually Content-Encoding: gzip. Every part declares an X-Chunk-Type and carries chunk-specific headers. Paths are base64-encoded in headers because filesystem bytes aren't guaranteed to be UTF-8.

HeaderDirectionPurpose
X-Export-Cursorclient → serverWhere to resume. Base64 of a JSON cursor; the server decodes it to a raw JSON string before handing it to a producer.
X-Cursorserver → client (per part)The new resume point after this chunk. Base64 of JSON.
X-Chunk-Typeserver → clientOne of sql · table_stats · progress · metadata · file · directory · symlink · index · index_batch · missing · error · completion.
X-File-Path / -Size / -Ctimeserver → clientPer-file metadata (path base64-encoded). X-First-Chunk / X-Last-Chunk flag multi-chunk files.
X-Statusserver → client (completion)complete or partial — the latter means a cursor was emitted and the client must request again.
Golden rule Producers work with JSON cursor strings. export.php handles base64 for the wire. Never hand a base64 string to a producer constructor — that boundary is a recurring gotcha in the codebase.

07Cursor-based reentrancy

A cursor is a JSON object that encodes complete state — enough to reconstruct exactly where a stream stopped, even if the data underneath shifted between requests. Each producer keeps its cursor as small as it can.

SQL dump cursor

The richest one: current table, PK columns, last PK values, offset, state-machine state, the in-flight row, batch counters, and an oversized-column queue (byte offsets only). Binary string values are wrapped as {"__binary__": base64} for JSON safety.

Column types are re-queried on resume, not stored — the schema can change between requests. If a table was dropped mid-migration, it resets gracefully.

File content cursor

Minimal: {phase, root, path, ctime, bytes} — the last emitted path, its ctime (to detect changes), and the byte offset within the current file. No traversal stack. On resume it re-sorts the path list and binary-searches back to position.

File index cursor

A depth-first stack: {stack: [{dir, after}, …]} — for each directory frame, the last entry processed. Resume re-scandirs and skips past after.

Keyset pagination, not OFFSET

For SQL, resumption uses keyset pagination: build_pk_where_clause() emits an expanded lexicographic comparison (pk₁, pk₂, …) > (last₁, last₂, …). This is O(1) per page regardless of table size. Tables with no primary key fall back to OFFSET pagination (slower, but correct). Oversized column values aren't carried in the cursor at all — only their byte offset — and are re-fetched on demand with SUBSTRING(), keeping the cursor tiny no matter how large a single cell is.

08Resource budgeting

The server's defense against being killed: stop voluntarily, just before the host's limits, and hand back a cursor.

Each non-preflight request builds a ResourceBudget with a start time, a max time (clamped 1–60s, default 5), and a memory ceiling derived from ini_get('memory_limit') times a threshold (clamped 0.1–0.95, default 0.8). Every streaming loop checks $budget->has_remaining() between chunks:

has_remaining():  (now - start) < max_time
              AND  memory_get_usage(true) < max_memory * threshold

When the budget runs out, the loop breaks, the completion chunk reports X-Status: partial, and the cursor rides along. endpoint_sql_chunk goes further: it derives a per-query MAX_EXECUTION_TIME MySQL hint (80% of the time budget) and caps statement size to 80% of the smaller of the client/server max_allowed_packet, so a single query can't overrun the budget or blow the packet limit.

Why this matters On a host with a 30-second max_execution_time and 256 MB memory limit, a naive dumper would get OOM-killed or time-fataled on a large table — losing all progress. Reprint instead returns 200 OK with partial status every few seconds and is asked again. Termination becomes a normal event, not a failure.

09AIMD adaptive tuning

The client doesn't guess how much to ask for. It borrows TCP's congestion-control idea — Additive Increase / Multiplicative Decrease — to converge on the largest request size the server can handle without timing out.

AdaptiveTuner tunes one "size" knob per endpoint, persisted in state.tuning so it survives restarts:

EndpointKnob (request param)Bounds · startWork metric
file_fetchchunk bytes (chunk_size)256 KB – 16 MB · 5 MBbytes processed
file_indexentries (batch_size)500 – 50,000 · 5,000entries processed
sql_chunkfragments (fragments_per_batch)100 – 5,000 · 1,000SQL bytes
flowchart TD
  R["record_result(endpoint, metrics)"]:::s
  R --> EMA["update throughput EMA
(α = 0.2)"]:::n EMA --> D{throughput_ratio
vs drop_ratio 0.9}:::q D -->|"ratio < 0.9 (got slower)"| DEC["MULTIPLICATIVE DECREASE
size × 0.7"]:::dec D -->|"ratio ≥ 0.9 (holding/faster)"| INC["ADDITIVE INCREASE
size + fixed step"]:::inc DEC --> CL["clamp to [min, max]"]:::n INC --> CL ERR["record_error() — 4xx/5xx/timeout"]:::err ERR --> BO["set error_backoff (3 reqs)
size × 0.5 immediately"]:::dec CL --> SLEEP["duty-cycle sleep:
space requests to a fraction
of server time"]:::n classDef s fill:#0c1322,stroke:#0c1322,color:#fff; classDef n fill:#f1f4fa,stroke:#cdd6e6,color:#37445c; classDef q fill:#fff4e6,stroke:#f0c182,color:#8a5410; classDef inc fill:#eafaf8,stroke:#7fd8cf,color:#0a6b62; classDef dec fill:#fdeef0,stroke:#f0b6bd,color:#a23440; classDef err fill:#fdeef0,stroke:#f0b6bd,color:#a23440;
Additive increase probes for more throughput; multiplicative decrease backs off fast when the server slows. A real error triggers an immediate halving and a few requests of backoff.

It only tunes on partial responses where real work happened, prefers server-reported time over wall-clock, and computes a duty-cycle sleep between requests so a fleet of importers doesn't synchronize and overwhelm the source.

10Crash recovery

Resumption isn't just for clean budget stops — it has to survive PHP fatals, OOM kills, SIGINT, and half-written files.

Partial file truncation

State tracks current_file + current_file_bytes (the size after the last saved cursor). On resume, if the file on disk is larger than that — meaning it crashed after writing but before saving — it ftruncates back to the last good byte and re-opens in append mode. The same logic protects db.sql via sql_bytes.

SQL stream crash recovery

In --sql-output=mysql mode, SQL is appended to a .sql-buffer file as it streams. If the transport dies before the completion chunk, the importer saves the cursor, sets status partial, exits 2, and reloads the buffer next run. The finally block deliberately doesn't throw about un-flushed SQL — that would mask the original error.

Atomic state writes

Every state/status/index file is written to a .tmp then renamed, so a crash mid-write never corrupts it. A corrupt .import-state.json on load is renamed aside and the importer starts fresh rather than crashing.

Signals & stall guards

SIGINT/SIGTERM flush the index, save state, then hard-kill the process (curl_close() during an active curl_exec() hangs). A consecutive-timeout guard gives up after 3 timeouts with no cursor progress, so it never loops forever against a dead server.


11Deep dive — MySQLDumpProducer

The SQL exporter is a finite-state machine that yields one fragment per next_sql_fragment() call — a CREATE TABLE, a batched multi-row INSERT (250 rows by default), or a special UPDATE…CONCAT for oversized rows. The whole machine can freeze and thaw on the cursor.

stateDiagram-v2
  [*] --> INIT
  INIT --> EMIT_HEADER
  EMIT_HEADER --> NEXT_TABLE
  NEXT_TABLE --> CREATE_TABLE: more tables
  NEXT_TABLE --> EMIT_FOOTER: no more tables
  CREATE_TABLE --> TABLE_HEADER
  TABLE_HEADER --> START_INSERT
  START_INSERT --> EMIT_ROW
  EMIT_ROW --> EMIT_ROW: accumulate up to batch_size
  EMIT_ROW --> START_INSERT: flush INSERT, more rows
  EMIT_ROW --> EMIT_OVERSIZED_UPDATE: row exceeds packet
  EMIT_OVERSIZED_UPDATE --> EMIT_ROW
  EMIT_ROW --> NEXT_TABLE: table drained
  EMIT_FOOTER --> FINISHED
  FINISHED --> [*]
      
Ten STATE_* constants. The cursor stores which state it's in plus everything needed to re-enter it on the next request.

Values travel as base64

Non-trivial column values are emitted wrapped in FROM_BASE64('…') (or CONVERT(FROM_BASE64('…') USING utf8mb4) for JSON/text). This sidesteps every escaping and character-set hazard in SQL transport — and, crucially, it's the hook the importer's URL rewriter latches onto. The dump remains compatible with standard MySQL import tools.

Oversized rows A single cell larger than the packet budget can't fit in any INSERT. The producer emits a base INSERT with a placeholder, then a queue of UPDATE … SET col = CONCAT(col, FROM_BASE64('…')) statements that append the value in budget-sized slices — re-fetched from the source with SUBSTRING() by byte offset so the cursor stays tiny.

12Deep dive — file sync & delta

A files-pull moves through three client-side stages, tracked in state.stage. The second run after a completed sync becomes a delta sync automatically.

flowchart LR
  subgraph idx["① index"]
    A["download_remote_index()
stream remote tree →
.import-remote-index.jsonl"]:::st A2["discover symlink targets
+ external-merge-sort
(sort by path)"]:::st A --> A2 end subgraph dff["② diff"] B["merge-join local vs remote
(both sorted by path)"]:::st end subgraph ftc["③ fetch"] C["download_files_from_list()
file_fetch endpoint →
--fs-root"]:::st end idx --> dff --> ftc B -. "checkpoint every 200 entries
{remote_offset, local_after}" .-> B classDef st fill:#eafaf8,stroke:#7fd8cf,color:#0a6b62;
The local index .import-index.jsonl is the memory of what's already on disk; the diff is what makes re-runs cheap.

The delta is a merge-join

diff_indexes_and_build_fetch_list() walks two sorted JSONL indexes with one pointer each, comparing paths:

ComparisonMeaningAction
local < remotepath gone from sourcedelete the local file + index entry
local == remotesame pathif ctime/size/type differ → queue for download
local > remote (or local exhausted)new remote pathqueue for download (unless preserve-local skips it)

A first-ever sync has an empty local index, so every remote entry becomes a download — a full pull. Progress is checkpointed every 200 entries ({remote_offset, local_after}) so even the diff phase is resumable. Index mutations land in .import-index-updates.jsonl and are merged into the canonical index only on success.

--filter — bring the site up before the uploads land

essential-files downloads code/config/themes/plugins and routes the uploads directory to a skipped list, then marks complete — so you can apply the DB and go live. skipped-earlier later drains that list. The uploads path comes from preflight (uploads.basedir).

Volatile files & preserve-local

If a file changes mid-stream (hash mismatch), it's recorded in .import-volatile-files.json rather than failing. With --on-fs-root-nonempty=preserve-local, anything already on disk — including any symlinked path component — is never overwritten, protecting shared host installs.


13Deep dive — URL rewriting on import

This is the cleverest subsystem. When a site's domain changes, every URL in the database must be rewritten — but those URLs are buried inside serialized PHP, JSON, and block markup where a naive find-and-replace would corrupt length prefixes and break the data. db-apply rewrites them in a single pass, structure-aware, and never calls unserialize().

flowchart TD
  SQL["SQL statement from db.sql"]:::in
  SQL --> PF{"cheap prefilters:
contains FROM_BASE64( ?
contains base64-of-'http' ?"}:::q PF -->|no| SKIP["pass through unchanged
(~280× faster on URL-free dumps)"]:::skip PF -->|yes| SCAN["FastInsertScanner / lexer →
Base64ValueScanner
locate + lazily decode each value"]:::n SCAN --> COL["map value → column
(block-markup columns hinted)"]:::n COL --> FMT{detect format of
decoded value}:::q FMT -->|"a: s: O: C:"| PHP["PhpSerializationProcessor
walk string leaves only"]:::php FMT -->|"starts { [ or quote"| JSON["JsonStringIterator
walk string leaves only"]:::json FMT -->|block_markup| BM["BlockMarkupUrlProcessor
HTML attrs, block JSON, CSS url()"]:::bm FMT -->|plain text| TXT["URLInTextProcessor"]:::txt PHP -->|recurse on each leaf| FMT JSON -->|recurse on each leaf| FMT PHP --> REENC["recompute s:N: length prefix"]:::out JSON --> REENC2["re-encode JSON
(unescaped slashes/unicode)"]:::out BM --> WRITE TXT --> WRITE REENC --> WRITE["re-base64 only changed values →
rebuilt SQL (or PDO-bound for SQLite)"]:::out REENC2 --> WRITE classDef in fill:#0c1322,stroke:#0c1322,color:#fff; classDef q fill:#fff4e6,stroke:#f0c182,color:#8a5410; classDef n fill:#f1f4fa,stroke:#cdd6e6,color:#37445c; classDef php fill:#f3effe,stroke:#c0adfb,color:#46329b; classDef json fill:#eafaf8,stroke:#7fd8cf,color:#0a6b62; classDef bm fill:#eef4ff,stroke:#9bb8f0,color:#1c3a6e; classDef txt fill:#eef4ff,stroke:#9bb8f0,color:#1c3a6e; classDef out fill:#eafaf8,stroke:#7fd8cf,color:#0a6b62; classDef skip fill:#f1f4fa,stroke:#cdd6e6,color:#7a8499;
Format detection is "conservative syntax gate, then try-and-fail." Serialized PHP and JSON recurse, so a URL nested three levels deep inside a serialized array of JSON still gets rewritten.

SqlStatementRewriter — the orchestrator

Owns the per-statement flow. Cheap prefilters reject URL-free statements (the four base64 prefixes aHR0 · dHA6 · dHBz · dHRw cover "http" at every byte alignment). A fast path recognizes the producer's exact INSERT shape with regex and skips the SQL lexer entirely; the slow path lexes once and shares the tokens. It binary-searches a column map so it knows which content type each value is.

PhpSerializationProcessor — the safety net

A cursor parser (no object instantiation, no __wakeup). It reads s:N:"…" by reading exactly N bytes, so embedded quotes and semicolons are safe, and on output it recomputes the byte-length prefix from the rewritten string. A URL that changes length still produces valid serialized data.

Supporting cast

  • StructuredDataUrlRewriter — pre-parses the --rewrite-url mappings once (a hoist that collapsed 94% of db-apply wall time under WASM PHP), detects format, and dispatches leaves to the right processor. Two LRU caches avoid re-rewriting identical values/URLs.
  • Base64ValueScanner — finds FROM_BASE64('…') occurrences (and the CONVERT(… USING utf8mb4) wrapper), decodes lazily, and rebuilds SQL re-encoding only the values that actually changed.
  • SQLitePreparedInsertBuilder — for SQLite targets, replaces values with positional placeholders and binds decoded bytes through PDO, so payload bytes never re-enter SQL text. Caches templates by row "shape."
  • DomainCollector — discovers the source site's own domains/CDN origins from media contexts only (image/video tags and media blocks), so external links in prose don't pollute the --rewrite-url suggestions.

14Deep dive — MySQL query streams

Before rewriting, db.sql must be split into individual statements as it streams in. There are two implementations with an identical contract, chosen for correctness-vs-speed.

WP_MySQL_Naive_Query_Stream

The correctness baseline. Spawns a fresh WP_MySQL_Lexer on every next_query() to find the next top-level semicolon. Accurate, but quadratic in statements per buffer.

WP_MySQL_FastQueryStream — default, ~27× faster

Replaces lexing with a tiny boundary scanner that only understands string literals, backtick identifiers, and comments, using strcspn() to skip "boring" stretches (base64 payloads) at libc speed. A persistent scan cursor resumes partial scans instead of re-scanning.

Graceful fallback The fast stream installs the naive parser and forwards to it the moment it hits something it can't safely handle (a 2 MB buffer with no top-level ;, or an unterminated construct at end-of-input). It logs the switch to the audit log and keeps bytes_consumed monotonic so resume still works. Speed by default, correctness as a guaranteed floor.

15Deep dive — hosts & runtimes

apply-runtime makes the downloaded files actually run. It cleanly separates "what was the source host doing?" from "what does my target server need?" — connected by a pure-data manifest.

flowchart LR
  PF["preflight data"]:::in
  PF --> DET["detect_host()
analyzers score 0–1"]:::n DET --> HA{best score ≥ 0.5}:::q HA -->|wpcloud| WC["WpcloudHostAnalyzer"]:::an HA -->|siteground| SG["SitegroundHostAnalyzer"]:::an HA -->|else| DF["DefaultHostAnalyzer"]:::an WC --> RM["RuntimeManifest
php_ini · constants · server_vars
routes · paths_to_remove · extra_directories · sqlite"]:::manifest SG --> RM DF --> RM RM --> AP{runtime_applier_for()}:::q AP -->|nginx-fpm| NG["NginxFpmApplier
runtime.php + nginx.conf"]:::ap AP -->|php-builtin| PB["PhpBuiltinApplier
runtime.php + start.sh"]:::ap AP -->|playground-cli| PG["PlaygroundCliApplier
runtime.php + blueprint.json"]:::ap classDef in fill:#0c1322,stroke:#0c1322,color:#fff; classDef n fill:#f1f4fa,stroke:#cdd6e6,color:#37445c; classDef q fill:#fff4e6,stroke:#f0c182,color:#8a5410; classDef an fill:#f3effe,stroke:#c0adfb,color:#46329b; classDef manifest fill:#eafaf8,stroke:#7fd8cf,color:#0a6b62; classDef ap fill:#eef4ff,stroke:#9bb8f0,color:#1c3a6e;
Adding a new source host or a new target server is a one-interface change that doesn't touch the other side.

The RuntimeManifest is the contract: php_ini directives to preserve, PHP constants and server_vars (with a {fs-root} placeholder resolved at apply time), declarative routes, paths_to_remove (production-only infra stripped after flattening), and extra_directories (dirs outside the WP root, e.g. from auto_prepend_file).

WpcloudHostAnalyzer

Detects WP Cloud / Atomic by the __wp__ directory, WP at doc_root/__wp__/, and the PRIVACY_MODEL env var. Strips the Memcached object-cache.php and wpcomsh mu-plugins, finds auto_prepend_file directories, and declares a thumbnail-generator route — because WP Cloud exports ship only full-size uploads, so missing -WxH sizes are generated on the fly with GD before WordPress even boots.

Plugin deactivation, derived from the manifest

Any paths_to_remove entry under wp-content/plugins/ also triggers removal from the active_plugins option at the end of db-apply — while the DB connection is still open. This avoids "plugin file does not exist" warnings without firing deactivation hooks into code that's already gone from disk. SiteGround strips sg-cachepress and sg-security the same way.

Two route handlers are inlined into the generated runtime.php: the WP Cloud thumbnail generator, and a remote upload proxy that streams missing uploads straight from the source while files-pull is still running — so the site looks complete before the media library finishes downloading.


16End to end — the pull pipeline

Pull (in lib/pull/class-pull.php) is the conductor. It composes the low-level commands into a single resumable pipeline, computing which stages are needed from the options you passed.

flowchart LR
  P["1 · Preflight"]:::s --> F["2 · Files
(files-pull)"]:::s F --> D["3 · Database
(db-pull)"]:::s D --> A["4 · Import
(db-apply +
URL rewrite)"]:::s A --> FL["5 · Flatten
(flat-docroot)"]:::opt FL --> RT["6 · Runtime
(apply-runtime)"]:::s RT --> ST["7 · Start
(launch server)"]:::s classDef s fill:#2f6df6,stroke:#2455c7,color:#fff; classDef opt fill:#eef4ff,stroke:#9bb8f0,color:#1c3a6e;
Stages 4–6 are conditional: db-apply only with a DB target, flat-docroot only with --flatten-to, start only when the runtime is launchable in-process.

Two mechanisms make it seamless:

  • run_until_complete() wraps each network stage in a loop (up to 1,000 iterations): if a stage returns partial (exit 2), it resets status and retries — turning the per-request resume protocol into a single smooth stage from the user's point of view.
  • Resume + delta — Pull reads the last completed stage from state.pull.stage and skips ahead. If the previous pull already completed, prepare_repull() sets up a delta: it keeps the local index and preflight, clears the transfer state, and deletes the dump + remote index so they're re-fetched. Re-running pull after success just syncs what changed.

17State files reference

All state lives as files in --state-dir, written atomically. Progress is computed client-side by reading them — keeping the protocol minimal. The two an integrator cares about are .import-state.json (the brain) and .import-status.json (pollable progress).

FileRole
.import-state.jsonThe canonical resumable state: active command, status, cursor, stage, cached preflight, AIMD tuning, per-phase bookmarks. Corrupt → renamed aside, start fresh.
.import-status.jsonFlat, pollable progress for a UI: step, steps, command, status, phase, error, ts, plus files_done / files_total during fetch.
.import-index.jsonlSorted local file index — the basis for delta detection (what's already on disk).
.import-remote-index.jsonlLatest remote tree snapshot from the server.
.import-download-list.jsonl / -skipped.jsonlFiles the diff decided to fetch / defer (under --filter).
.import-volatile-files.jsonFiles that changed mid-stream, with a per-path counter.
.import-audit.logAppend-only event log (downloads, deletes, volatile detections, errors). Never rotated.
db.sql / .sql-bufferThe downloaded dump / the in-flight buffer for --sql-output=mysql crash recovery.

18Distribution & the Playground UI

Two artifacts ship, plus a browser-based wizard that runs the importer entirely client-side.

reprint.phar

Built by Box (bin/build-reprint-phar.sh + box.json). The version comes from git tags into a baked-in VERSION file. The bundled sqlite-database-integration submodule is compiled in, so the phar is self-contained and needs no MySQL on the target. A single CLI guard keyed on IMPORTER_PHAR_ENTRY / IMPORTER_WRAPPER_ENTRY / IMPORTER_WEB_ENTRY lets the same file run as a phar, a source wrapper, or an embedded include.

reprint-exporter-wp.zip

The WordPress plugin, with the exporter Composer package vendored in. stamp-plugin-version.sh stamps the plugin version into index.php + lib.php. The admin UI (Site_Export_Plugin) manages the HMAC secret via a WP option exposed over REST, so platforms can provision it programmatically.

The Playground wizard (reprint-ui/)

A WordPress.com → Playground handoff that runs the migration in the browser and never touches site bytes server-side. It's a stateless PHP app (signed, encrypted cookies — no sessions): the user authorizes via WP.com OAuth, the server enables the exporter and rotates a 60-minute HMAC secret on the chosen Atomic site, then hands off to playground.wordpress.net with a generated Blueprint. Inside Playground, reprint-import.php includes the same reprint.phar and runs pull … --target-engine=sqlite --runtime=none — the browser is the network transport, Playground is the server, and a uploads-proxy mu-plugin streams missing media from the source until the sync completes.


19Key takeaways

⏸️

Resumption is the spine

Cursors, exit code 2, budgets, atomic state, truncation-on-resume — every layer assumes any request can stop at any moment.

🔌

One narrow interface

Everything goes through one HMAC-signed multipart endpoint on a WordPress plugin — the lowest common denominator that works on locked-down hosts.

🧩

Pluggable seams

PDO adapters, host analyzers ↔ runtime appliers (via a data manifest), and naive ↔ fast query streams are all swappable behind stable contracts.

📈

Adapts to the host

AIMD tuning and resource budgeting mean the same binary runs well on a 256 MB shared host and a beefy server without configuration.

🔒

Structure-preserving rewrites

URL rewriting walks serialized PHP / JSON / block markup without unserialize(), recomputing length prefixes — correctness over shortcuts.

🚀

Multiple targets, one path

MySQL, SQLite, nginx, PHP built-in server, and browser Playground all reuse the same import engine.

Where to read next, in the source Start with packages/reprint-exporter/src/class-http-server.php (the router) and export.php (endpoints) for the server; packages/reprint-importer/src/lib/pull/class-pull.php for the orchestration, then ImportClient in import.php for the engine. The README.md documents the CLI contract and state-file formats in full.