How Reprint works
Reprint clones an entire WordPress site — files, database, and server configuration — over plain HTTP, in resumable streaming chunks designed to survive the tightest shared-hosting limits. This document maps every block of the repo and the patterns that hold them together.
01What Reprint is
Reprint copies a live WordPress site from a source (a remote site you don't fully control — often locked-down shared hosting) to a target (your laptop, or another host). The whole transfer happens over ordinary HTTP requests against a single WordPress plugin endpoint, so it works even where SSH, direct DB access, and raw PHP execution are blocked.
One command
reprint.phar pull https://example.com pulls files + DB + config and starts a local copy you can open in a browser.
Resumable everywhere
Every transfer is broken into chunks with a JSON cursor. Interrupt it, re-run the same command, it continues from the last byte.
Memory & time aware
The server stops a request before hitting the host's memory/time limit and hands back a cursor — never an OOM kill.
The architecture is a textbook producer–consumer split with a streaming pipe between them:
| Role | Lives in | What it does |
|---|---|---|
| Producer (server) | packages/reprint-exporter, shipped as the reprint-exporter-wp WordPress plugin | Answers authenticated HTTP requests by streaming SQL dump fragments and file contents, pausing on a budget and emitting a resume cursor. |
| Consumer (client) | packages/reprint-importer, shipped as reprint.phar | A CLI that drives the migration: indexes, diffs, downloads, rewrites URLs, applies the DB, and generates runtime config. |
| The pipe | HTTP | multipart/mixed, gzip-compressed, HMAC-signed, with cursors carried in headers. |
02System overview
At the highest level there are two PHP processes that never share a filesystem or a database — they only exchange HTTP requests. The source runs inside WordPress; the target is a standalone CLI.
flowchart LR
subgraph SRC["🖥️ MIGRATION SOURCE — remote WordPress site"]
direction TB
WP["WordPress front controller
?reprint-api"]:::wp
PLG["reprint-exporter-wp
plugin shell · HMAC auth"]:::plg
HTTPS["Site_Export_HTTP_Server
router + ResourceBudget"]:::core
MDP["MySQLDumpProducer"]:::prod
FTP["FileTreeProducer"]:::prod
DB[("MySQL / SQLite")]:::store
FS[("Filesystem")]:::store
WP --> PLG --> HTTPS
HTTPS --> MDP --> DB
HTTPS --> FTP --> FS
end
subgraph TGT["💻 MIGRATION TARGET — your machine / a host"]
direction TB
CLI["reprint.phar — CLI dispatch"]:::cli
IC["ImportClient
(state machine, ~9k LOC)"]:::core
MSP["MultipartStreamParser"]:::prod
REW["URL Rewriter +
Query Stream"]:::prod
HOST["Host analyzer →
RuntimeManifest → applier"]:::prod
OUT[("--fs-root files
db.sql / target DB
runtime.php")]:::store
CLI --> IC --> MSP
IC --> REW --> OUT
IC --> HOST --> OUT
end
HTTPS == "multipart/mixed · gzip · X-Cursor
HMAC-SHA256 signed" ==> MSP
IC -. "GET/POST ?endpoint=… + X-Export-Cursor" .-> HTTPS
classDef wp fill:#eef4ff,stroke:#9bb8f0,color:#1c3a6e;
classDef plg fill:#f3effe,stroke:#c0adfb,color:#46329b;
classDef core fill:#0c1322,stroke:#0c1322,color:#fff;
classDef prod fill:#eafaf8,stroke:#7fd8cf,color:#0a6b62;
classDef cli fill:#fff4e6,stroke:#f0c182,color:#8a5410;
classDef store fill:#f1f4fa,stroke:#cdd6e6,color:#37445c;
03Repository map
The repo is a monorepo: two Composer packages (exporter + importer), a WordPress plugin that bundles the exporter, a Playground-based web UI, and the build tooling that produces the release artifacts.
reprint/
├── packages/
│ ├── reprint-exporter/src/ # PRODUCER (server)
│ │ ├── export.php # endpoint functions + streaming engine (~3.8k LOC)
│ │ ├── class-http-server.php # Site_Export_HTTP_Server — request router
│ │ ├── class-mysql-dump-producer.php # SQL dump state machine (~1.5k LOC)
│ │ ├── class-file-tree-producer.php # file content streamer
│ │ ├── class-hmac-server.php / -client.php # auth
│ │ └── class-sqlite-driver-pdo.php / -wpdb-driver-pdo.php / -pdo-polyfill.php # DB fallbacks
│ │
│ └── reprint-importer/src/ # CONSUMER (client)
│ ├── import.php # the whole CLI engine (~12k LOC)
│ └── lib/
│ ├── pull/class-pull.php # the high-level pull orchestrator
│ ├── url-rewrite/ # the db-apply URL rewriting pipeline
│ ├── host/ # source-host analyzers + RuntimeManifest
│ ├── target-runtime/ # runtime appliers (nginx / php-builtin / playground)
│ ├── mysql-query-stream/ # statement splitters (naive + fast)
│ └── external-merge-sort.php # on-disk sort for huge index files
│
├── reprint-exporter-wp/ # WordPress plugin distribution (index.php + lib.php + admin UI)
├── reprint-ui/ # WP.com → Playground browser wizard
├── lib/sqlite-database-integration/ # git submodule: WordPress' MySQL→SQLite driver
├── importer/import.php # thin wrapper to run the importer from source
├── bin/ # build-reprint-phar.sh, phar-stub.php, stamp-plugin-version.sh
└── box.json # Box config — compiles reprint.phar
import.php is a single ~12,000-line file built almost entirely around one class, ImportClient. That's deliberate — the phar must run as a self-contained script on hosts where you can't composer install. The supporting lib/ subsystems are pulled in by the build.
04Export side — the producer (server)
The export engine lives in packages/reprint-exporter and runs inside WordPress via the reprint-exporter-wp plugin. Its job: authenticate a request, pick the right producer, and stream a chunked response that can pause and resume.
Why a WordPress plugin, and why ?reprint-api
Many shared hosts (SiteGround, GoDaddy, WP Cloud) block direct PHP execution inside wp-content/plugins/ at the web-server level — a 403 before PHP ever runs. So the plugin doesn't expose its own URL. Instead it routes through WordPress's own front controller, which hosts never block:
reprint-exporter-wp/index.phpchecks$_GET['reprint-api'](canonical) or the legacy alias$_GET['site-export-api']at plugin-load time — the earliest interception point a normal plugin gets — and calls_site_export_handle_api_request()thenexit.- If it's not an API request, it loads the admin settings UI instead (
wordpress/site-export.php).
The request lifecycle
sequenceDiagram autonumber participant C as ImportClient (target) participant WP as WordPress + plugin participant H as Site_Export_HTTP_Server participant A as Site_Export_HMAC_Server participant E as endpoint_* (export.php) participant P as Producer C->>WP: GET /?reprint-api&endpoint=sql_chunk
X-Auth-* headers, X-Export-Cursor WP->>H: handle_cors_headers_and_terminate_on_options('*') Note over H: OPTIONS short-circuits
(preflight has no creds) WP->>A: _site_export_verify_hmac(secret) A-->>WP: timestamp ±300s? nonce ok?
HMAC + content-hash match? Note over A: fail → 403/503 JSON, exit WP->>H: serve(['default_directory'=>ABSPATH]) H->>H: parse_http_config() · normalize_config()
base64-decode X-Export-Cursor → JSON H->>H: build ResourceBudget (time + memory) H->>E: dispatch(endpoint, config, budget) E->>P: new Producer(pdo/paths, json_cursor) loop until producer finished OR budget exhausted P-->>E: next fragment / chunk E-->>C: multipart part — X-Chunk-Type, X-Cursor, gzip end E-->>C: completion part — X-Status: complete | partial
The five endpoints
The client selects one with the endpoint parameter. They're registered in Site_Export_HTTP_Server::default_handlers():
| Endpoint | Handler | What it streams |
|---|---|---|
| preflight | endpoint_preflight | Plain JSON (not multipart): WP version, PHP/MySQL versions, memory limits, FS access, DB connectivity, plugins, themes, directory layout, multisite. Every other command refuses to run without it. |
| db_index | endpoint_db_index | Table metadata (name, estimated rows, data/index bytes, engine, collation) from INFORMATION_SCHEMA, paginated by TABLE_NAME > :last. |
| sql_chunk | endpoint_sql_chunk | SQL dump fragments produced by MySQLDumpProducer, gzip multipart. |
| file_index | endpoint_file_index | A depth-first directory listing in JSON batches: {path, ctime, size, type, target}. Carries its own stack-based cursor. |
| file_fetch | endpoint_file_fetch | The bytes of a client-supplied list of paths, via FileTreeProducer. |
endpoint_file_index guards against the pathological symlink layouts of WP Cloud Atomic — e.g. /srv/htdocs/srv → /srv (an infinite cycle) or /wordpress/ and /srv/htdocs/wordpress/ resolving to the same place. should_skip_index_root() compares each directory's realpath() against the scheduled roots and skips duplicates/parents before traversal recurses into them.
Streaming + database access
GzipOutputStream
Output buffering, zlib.output_compression, and proxy buffering (X-Accel-Buffering: no) are all disabled so a custom incremental gzip stream controls flushing: deflate_add(ZLIB_NO_FLUSH) while writing a part, ZLIB_SYNC_FLUSH at part boundaries. This keeps memory flat regardless of payload size.
Errors become chunks, not 500s
Once the gzip stream is open you can't send an HTTP error code. A global $streaming_context lets the error / exception / shutdown handlers inject an X-Chunk-Type: error part mid-stream instead. The completion chunk is itself wrapped in try/catch so a broken stream is logged, never fatal.
The PDO fallback ladder
The producers only ever call PDO-shaped methods (query, prepare, quote, fetch, bindValue, execute). create_db_connection() hands them one of three objects with an identical surface, so the producer code never branches:
flowchart TD
Q{What does the host have?}:::q
Q -->|"SQLite site
(SQLITE_DB_DROPIN_VERSION)"| S["SqliteDriverPDO
wraps WP_SQLite_Driver"]:::a
Q -->|"no ext-pdo_mysql"| W["WpdbDriverPDO
wraps global \$wpdb"]:::a
Q -->|"ext-pdo_mysql present"| R["real PDO
mysql DSN, utf8mb4"]:::a
S --> M["MySQLDumpProducer
(same code path either way)"]:::m
W --> M
R --> M
PP["class-pdo-polyfill.php
defines PDO / PDOStatement / PDOException
so PDO:: constants don't fatal on PDO-less hosts"]:::p
PP -.underpins.-> S
PP -.underpins.-> W
classDef q fill:#fff4e6,stroke:#f0c182,color:#8a5410;
classDef a fill:#eafaf8,stroke:#7fd8cf,color:#0a6b62;
classDef m fill:#0c1322,stroke:#0c1322,color:#fff;
classDef p fill:#f3effe,stroke:#c0adfb,color:#46329b;
wpdb adapter bridges the gap.05Import side — the consumer (client)
The importer is a single CLI (reprint.phar) built around the ImportClient class. It has two layers: a set of low-level commands (one network concern each), and a high-level pull orchestrator that chains them.
Command dispatch
The bottom of import.php parses argv, applies legacy aliases (files-sync→files-pull, db-sync→db-pull, …), constructs new ImportClient($url, $stateDir, $fsRoot), and calls run($options), which routes to a run_<command>() method. EXIT_AFTER_IMPORT gates whether the process exits or returns (the latter for the Playground embedder).
| Command | Level | Role |
|---|---|---|
pull | high | The full resumable clone — chains everything below into one pipeline. |
install-exporter | high | Prints the plugin download URL + install steps. No network. |
preflight / preflight-assert | low | Probe the remote, cache the result, print JSON / a human pass-fail. |
files-index | low | Stream the remote directory tree to .import-remote-index.jsonl. No file contents. |
files-pull | low | Index → diff → download changed files into --fs-root. Delta on re-run. |
db-index | low | Table metadata → db-tables.jsonl. |
db-pull | low | Stream the SQL dump to db.sql, stdout, or a live MySQL connection. |
db-apply | low | Execute db.sql into MySQL/SQLite, rewriting URLs in one pass. |
db-domains | low | List domains discovered in the dump (for choosing --rewrite-url maps). |
flat-docroot | low | Reassemble a non-standard layout into a normal WP docroot via symlinks. |
apply-runtime | low | Generate server config (runtime.php + nginx.conf / start.sh) from preflight data. |
Exit codes are the control flow
Because any request can be cut short, the low-level commands don't loop internally. They lean on three exit codes and let the caller (a shell loop, or the pull orchestrator) re-invoke:
0 — complete
state.status === 'complete'. The stage is done.
2 — partial
Progress was made, call me again. Cursor is saved; the next run resumes.
1 — fatal
An exception bubbled to the bootstrap; a JSON error is written to stderr.
The streaming HTTP client
One primitive does all the networking: ImportClient::fetch_streaming(). It's built on cURL with deliberate choices for hostile networks:
- No total timeout — a stream can legitimately run 20+ minutes. Instead
CURLOPT_LOW_SPEED_LIMIT=1/LOW_SPEED_TIME=300detect a genuinely stalled connection. - Cursor redundancy — the cursor goes out both as the
X-Export-Cursorheader and acursorquery param, because some proxies strip custom headers. - Browser-mimicking headers — a real-looking
User-AgentandSec-Fetch-*headers, because some WAFs block requests that carry custom auth headers under a non-browser UA. - Actionable diagnostics —
diagnose_http_error()turns redirects, WAF HTML pages, and JSON error payloads into human messages, and distinguishes a retryable timeout from a hard failure.
MultipartStreamParser — incremental, never buffered
cURL's WRITEFUNCTION feeds bytes into MultipartStreamParser as they arrive. It's a three-state machine — BOUNDARY → HEADERS → BODY — that emits two events: streaming body bytes (large files go straight to disk) and a complete event at each part boundary. It keeps a small tail when a boundary might be split across chunks, and a 64 MB buffer cap catches malformed responses (a WAF page with no boundary).
06The wire protocol
Two design decisions define the protocol: how requests are authenticated, and how responses are framed so they can be split, paused, and resumed.
HMAC authentication
There are no cookies or sessions. Every request is signed with a shared secret using a scheme that survives cURL's unpredictable multipart framing:
signature = HMAC-SHA256( nonce + timestamp + SHA256(body) , secret ) // hex
The signature covers a hash of the body (sent as X-Auth-Content-Hash), not the raw bytes — so for file uploads the client signs the logical content the server can independently reconstruct, regardless of how cURL frames the multipart request. Site_Export_HMAC_Server::verify_globals() checks all four X-Auth-* headers are present, the timestamp is within ±300s (replay window), the nonce is ≥16 chars, and runs every comparison through hash_equals() (constant-time). The secret comes from secret.php (file override, takes precedence) or the site_export_secret WordPress option (set via the admin UI / REST).
multipart/mixed framing
Responses are Content-Type: multipart/mixed; boundary="boundary-<32 hex>", usually Content-Encoding: gzip. Every part declares an X-Chunk-Type and carries chunk-specific headers. Paths are base64-encoded in headers because filesystem bytes aren't guaranteed to be UTF-8.
| Header | Direction | Purpose |
|---|---|---|
X-Export-Cursor | client → server | Where to resume. Base64 of a JSON cursor; the server decodes it to a raw JSON string before handing it to a producer. |
X-Cursor | server → client (per part) | The new resume point after this chunk. Base64 of JSON. |
X-Chunk-Type | server → client | One of sql · table_stats · progress · metadata · file · directory · symlink · index · index_batch · missing · error · completion. |
X-File-Path / -Size / -Ctime | server → client | Per-file metadata (path base64-encoded). X-First-Chunk / X-Last-Chunk flag multi-chunk files. |
X-Status | server → client (completion) | complete or partial — the latter means a cursor was emitted and the client must request again. |
export.php handles base64 for the wire. Never hand a base64 string to a producer constructor — that boundary is a recurring gotcha in the codebase.
07Cursor-based reentrancy
A cursor is a JSON object that encodes complete state — enough to reconstruct exactly where a stream stopped, even if the data underneath shifted between requests. Each producer keeps its cursor as small as it can.
SQL dump cursor
The richest one: current table, PK columns, last PK values, offset, state-machine state, the in-flight row, batch counters, and an oversized-column queue (byte offsets only). Binary string values are wrapped as {"__binary__": base64} for JSON safety.
Column types are re-queried on resume, not stored — the schema can change between requests. If a table was dropped mid-migration, it resets gracefully.
File content cursor
Minimal: {phase, root, path, ctime, bytes} — the last emitted path, its ctime (to detect changes), and the byte offset within the current file. No traversal stack. On resume it re-sorts the path list and binary-searches back to position.
File index cursor
A depth-first stack: {stack: [{dir, after}, …]} — for each directory frame, the last entry processed. Resume re-scandirs and skips past after.
Keyset pagination, not OFFSET
For SQL, resumption uses keyset pagination: build_pk_where_clause() emits an expanded lexicographic comparison (pk₁, pk₂, …) > (last₁, last₂, …). This is O(1) per page regardless of table size. Tables with no primary key fall back to OFFSET pagination (slower, but correct). Oversized column values aren't carried in the cursor at all — only their byte offset — and are re-fetched on demand with SUBSTRING(), keeping the cursor tiny no matter how large a single cell is.
08Resource budgeting
The server's defense against being killed: stop voluntarily, just before the host's limits, and hand back a cursor.
Each non-preflight request builds a ResourceBudget with a start time, a max time (clamped 1–60s, default 5), and a memory ceiling derived from ini_get('memory_limit') times a threshold (clamped 0.1–0.95, default 0.8). Every streaming loop checks $budget->has_remaining() between chunks:
has_remaining(): (now - start) < max_time
AND memory_get_usage(true) < max_memory * threshold
When the budget runs out, the loop breaks, the completion chunk reports X-Status: partial, and the cursor rides along. endpoint_sql_chunk goes further: it derives a per-query MAX_EXECUTION_TIME MySQL hint (80% of the time budget) and caps statement size to 80% of the smaller of the client/server max_allowed_packet, so a single query can't overrun the budget or blow the packet limit.
max_execution_time and 256 MB memory limit, a naive dumper would get OOM-killed or time-fataled on a large table — losing all progress. Reprint instead returns 200 OK with partial status every few seconds and is asked again. Termination becomes a normal event, not a failure.
09AIMD adaptive tuning
The client doesn't guess how much to ask for. It borrows TCP's congestion-control idea — Additive Increase / Multiplicative Decrease — to converge on the largest request size the server can handle without timing out.
AdaptiveTuner tunes one "size" knob per endpoint, persisted in state.tuning so it survives restarts:
| Endpoint | Knob (request param) | Bounds · start | Work metric |
|---|---|---|---|
file_fetch | chunk bytes (chunk_size) | 256 KB – 16 MB · 5 MB | bytes processed |
file_index | entries (batch_size) | 500 – 50,000 · 5,000 | entries processed |
sql_chunk | fragments (fragments_per_batch) | 100 – 5,000 · 1,000 | SQL bytes |
flowchart TD R["record_result(endpoint, metrics)"]:::s R --> EMA["update throughput EMA
(α = 0.2)"]:::n EMA --> D{throughput_ratio
vs drop_ratio 0.9}:::q D -->|"ratio < 0.9 (got slower)"| DEC["MULTIPLICATIVE DECREASE
size × 0.7"]:::dec D -->|"ratio ≥ 0.9 (holding/faster)"| INC["ADDITIVE INCREASE
size + fixed step"]:::inc DEC --> CL["clamp to [min, max]"]:::n INC --> CL ERR["record_error() — 4xx/5xx/timeout"]:::err ERR --> BO["set error_backoff (3 reqs)
size × 0.5 immediately"]:::dec CL --> SLEEP["duty-cycle sleep:
space requests to a fraction
of server time"]:::n classDef s fill:#0c1322,stroke:#0c1322,color:#fff; classDef n fill:#f1f4fa,stroke:#cdd6e6,color:#37445c; classDef q fill:#fff4e6,stroke:#f0c182,color:#8a5410; classDef inc fill:#eafaf8,stroke:#7fd8cf,color:#0a6b62; classDef dec fill:#fdeef0,stroke:#f0b6bd,color:#a23440; classDef err fill:#fdeef0,stroke:#f0b6bd,color:#a23440;
It only tunes on partial responses where real work happened, prefers server-reported time over wall-clock, and computes a duty-cycle sleep between requests so a fleet of importers doesn't synchronize and overwhelm the source.
10Crash recovery
Resumption isn't just for clean budget stops — it has to survive PHP fatals, OOM kills, SIGINT, and half-written files.
Partial file truncation
State tracks current_file + current_file_bytes (the size after the last saved cursor). On resume, if the file on disk is larger than that — meaning it crashed after writing but before saving — it ftruncates back to the last good byte and re-opens in append mode. The same logic protects db.sql via sql_bytes.
SQL stream crash recovery
In --sql-output=mysql mode, SQL is appended to a .sql-buffer file as it streams. If the transport dies before the completion chunk, the importer saves the cursor, sets status partial, exits 2, and reloads the buffer next run. The finally block deliberately doesn't throw about un-flushed SQL — that would mask the original error.
Atomic state writes
Every state/status/index file is written to a .tmp then renamed, so a crash mid-write never corrupts it. A corrupt .import-state.json on load is renamed aside and the importer starts fresh rather than crashing.
Signals & stall guards
SIGINT/SIGTERM flush the index, save state, then hard-kill the process (curl_close() during an active curl_exec() hangs). A consecutive-timeout guard gives up after 3 timeouts with no cursor progress, so it never loops forever against a dead server.
11Deep dive — MySQLDumpProducer
The SQL exporter is a finite-state machine that yields one fragment per next_sql_fragment() call — a CREATE TABLE, a batched multi-row INSERT (250 rows by default), or a special UPDATE…CONCAT for oversized rows. The whole machine can freeze and thaw on the cursor.
stateDiagram-v2
[*] --> INIT
INIT --> EMIT_HEADER
EMIT_HEADER --> NEXT_TABLE
NEXT_TABLE --> CREATE_TABLE: more tables
NEXT_TABLE --> EMIT_FOOTER: no more tables
CREATE_TABLE --> TABLE_HEADER
TABLE_HEADER --> START_INSERT
START_INSERT --> EMIT_ROW
EMIT_ROW --> EMIT_ROW: accumulate up to batch_size
EMIT_ROW --> START_INSERT: flush INSERT, more rows
EMIT_ROW --> EMIT_OVERSIZED_UPDATE: row exceeds packet
EMIT_OVERSIZED_UPDATE --> EMIT_ROW
EMIT_ROW --> NEXT_TABLE: table drained
EMIT_FOOTER --> FINISHED
FINISHED --> [*]
STATE_* constants. The cursor stores which state it's in plus everything needed to re-enter it on the next request.Values travel as base64
Non-trivial column values are emitted wrapped in FROM_BASE64('…') (or CONVERT(FROM_BASE64('…') USING utf8mb4) for JSON/text). This sidesteps every escaping and character-set hazard in SQL transport — and, crucially, it's the hook the importer's URL rewriter latches onto. The dump remains compatible with standard MySQL import tools.
UPDATE … SET col = CONCAT(col, FROM_BASE64('…')) statements that append the value in budget-sized slices — re-fetched from the source with SUBSTRING() by byte offset so the cursor stays tiny.
12Deep dive — file sync & delta
A files-pull moves through three client-side stages, tracked in state.stage. The second run after a completed sync becomes a delta sync automatically.
flowchart LR
subgraph idx["① index"]
A["download_remote_index()
stream remote tree →
.import-remote-index.jsonl"]:::st
A2["discover symlink targets
+ external-merge-sort
(sort by path)"]:::st
A --> A2
end
subgraph dff["② diff"]
B["merge-join local vs remote
(both sorted by path)"]:::st
end
subgraph ftc["③ fetch"]
C["download_files_from_list()
file_fetch endpoint →
--fs-root"]:::st
end
idx --> dff --> ftc
B -. "checkpoint every 200 entries
{remote_offset, local_after}" .-> B
classDef st fill:#eafaf8,stroke:#7fd8cf,color:#0a6b62;
.import-index.jsonl is the memory of what's already on disk; the diff is what makes re-runs cheap.The delta is a merge-join
diff_indexes_and_build_fetch_list() walks two sorted JSONL indexes with one pointer each, comparing paths:
| Comparison | Meaning | Action |
|---|---|---|
local < remote | path gone from source | delete the local file + index entry |
local == remote | same path | if ctime/size/type differ → queue for download |
local > remote (or local exhausted) | new remote path | queue for download (unless preserve-local skips it) |
A first-ever sync has an empty local index, so every remote entry becomes a download — a full pull. Progress is checkpointed every 200 entries ({remote_offset, local_after}) so even the diff phase is resumable. Index mutations land in .import-index-updates.jsonl and are merged into the canonical index only on success.
--filter — bring the site up before the uploads land
essential-files downloads code/config/themes/plugins and routes the uploads directory to a skipped list, then marks complete — so you can apply the DB and go live. skipped-earlier later drains that list. The uploads path comes from preflight (uploads.basedir).
Volatile files & preserve-local
If a file changes mid-stream (hash mismatch), it's recorded in .import-volatile-files.json rather than failing. With --on-fs-root-nonempty=preserve-local, anything already on disk — including any symlinked path component — is never overwritten, protecting shared host installs.
13Deep dive — URL rewriting on import
This is the cleverest subsystem. When a site's domain changes, every URL in the database must be rewritten — but those URLs are buried inside serialized PHP, JSON, and block markup where a naive find-and-replace would corrupt length prefixes and break the data. db-apply rewrites them in a single pass, structure-aware, and never calls unserialize().
flowchart TD
SQL["SQL statement from db.sql"]:::in
SQL --> PF{"cheap prefilters:
contains FROM_BASE64( ?
contains base64-of-'http' ?"}:::q
PF -->|no| SKIP["pass through unchanged
(~280× faster on URL-free dumps)"]:::skip
PF -->|yes| SCAN["FastInsertScanner / lexer →
Base64ValueScanner
locate + lazily decode each value"]:::n
SCAN --> COL["map value → column
(block-markup columns hinted)"]:::n
COL --> FMT{detect format of
decoded value}:::q
FMT -->|"a: s: O: C:"| PHP["PhpSerializationProcessor
walk string leaves only"]:::php
FMT -->|"starts { [ or quote"| JSON["JsonStringIterator
walk string leaves only"]:::json
FMT -->|block_markup| BM["BlockMarkupUrlProcessor
HTML attrs, block JSON, CSS url()"]:::bm
FMT -->|plain text| TXT["URLInTextProcessor"]:::txt
PHP -->|recurse on each leaf| FMT
JSON -->|recurse on each leaf| FMT
PHP --> REENC["recompute s:N: length prefix"]:::out
JSON --> REENC2["re-encode JSON
(unescaped slashes/unicode)"]:::out
BM --> WRITE
TXT --> WRITE
REENC --> WRITE["re-base64 only changed values →
rebuilt SQL (or PDO-bound for SQLite)"]:::out
REENC2 --> WRITE
classDef in fill:#0c1322,stroke:#0c1322,color:#fff;
classDef q fill:#fff4e6,stroke:#f0c182,color:#8a5410;
classDef n fill:#f1f4fa,stroke:#cdd6e6,color:#37445c;
classDef php fill:#f3effe,stroke:#c0adfb,color:#46329b;
classDef json fill:#eafaf8,stroke:#7fd8cf,color:#0a6b62;
classDef bm fill:#eef4ff,stroke:#9bb8f0,color:#1c3a6e;
classDef txt fill:#eef4ff,stroke:#9bb8f0,color:#1c3a6e;
classDef out fill:#eafaf8,stroke:#7fd8cf,color:#0a6b62;
classDef skip fill:#f1f4fa,stroke:#cdd6e6,color:#7a8499;
SqlStatementRewriter — the orchestrator
Owns the per-statement flow. Cheap prefilters reject URL-free statements (the four base64 prefixes aHR0 · dHA6 · dHBz · dHRw cover "http" at every byte alignment). A fast path recognizes the producer's exact INSERT shape with regex and skips the SQL lexer entirely; the slow path lexes once and shares the tokens. It binary-searches a column map so it knows which content type each value is.
PhpSerializationProcessor — the safety net
A cursor parser (no object instantiation, no __wakeup). It reads s:N:"…" by reading exactly N bytes, so embedded quotes and semicolons are safe, and on output it recomputes the byte-length prefix from the rewritten string. A URL that changes length still produces valid serialized data.
Supporting cast
StructuredDataUrlRewriter— pre-parses the--rewrite-urlmappings once (a hoist that collapsed 94% of db-apply wall time under WASM PHP), detects format, and dispatches leaves to the right processor. Two LRU caches avoid re-rewriting identical values/URLs.Base64ValueScanner— findsFROM_BASE64('…')occurrences (and theCONVERT(… USING utf8mb4)wrapper), decodes lazily, and rebuilds SQL re-encoding only the values that actually changed.SQLitePreparedInsertBuilder— for SQLite targets, replaces values with positional placeholders and binds decoded bytes through PDO, so payload bytes never re-enter SQL text. Caches templates by row "shape."DomainCollector— discovers the source site's own domains/CDN origins from media contexts only (image/video tags and media blocks), so external links in prose don't pollute the--rewrite-urlsuggestions.
14Deep dive — MySQL query streams
Before rewriting, db.sql must be split into individual statements as it streams in. There are two implementations with an identical contract, chosen for correctness-vs-speed.
WP_MySQL_Naive_Query_Stream
The correctness baseline. Spawns a fresh WP_MySQL_Lexer on every next_query() to find the next top-level semicolon. Accurate, but quadratic in statements per buffer.
WP_MySQL_FastQueryStream — default, ~27× faster
Replaces lexing with a tiny boundary scanner that only understands string literals, backtick identifiers, and comments, using strcspn() to skip "boring" stretches (base64 payloads) at libc speed. A persistent scan cursor resumes partial scans instead of re-scanning.
;, or an unterminated construct at end-of-input). It logs the switch to the audit log and keeps bytes_consumed monotonic so resume still works. Speed by default, correctness as a guaranteed floor.
15Deep dive — hosts & runtimes
apply-runtime makes the downloaded files actually run. It cleanly separates "what was the source host doing?" from "what does my target server need?" — connected by a pure-data manifest.
flowchart LR PF["preflight data"]:::in PF --> DET["detect_host()
analyzers score 0–1"]:::n DET --> HA{best score ≥ 0.5}:::q HA -->|wpcloud| WC["WpcloudHostAnalyzer"]:::an HA -->|siteground| SG["SitegroundHostAnalyzer"]:::an HA -->|else| DF["DefaultHostAnalyzer"]:::an WC --> RM["RuntimeManifest
php_ini · constants · server_vars
routes · paths_to_remove · extra_directories · sqlite"]:::manifest SG --> RM DF --> RM RM --> AP{runtime_applier_for()}:::q AP -->|nginx-fpm| NG["NginxFpmApplier
runtime.php + nginx.conf"]:::ap AP -->|php-builtin| PB["PhpBuiltinApplier
runtime.php + start.sh"]:::ap AP -->|playground-cli| PG["PlaygroundCliApplier
runtime.php + blueprint.json"]:::ap classDef in fill:#0c1322,stroke:#0c1322,color:#fff; classDef n fill:#f1f4fa,stroke:#cdd6e6,color:#37445c; classDef q fill:#fff4e6,stroke:#f0c182,color:#8a5410; classDef an fill:#f3effe,stroke:#c0adfb,color:#46329b; classDef manifest fill:#eafaf8,stroke:#7fd8cf,color:#0a6b62; classDef ap fill:#eef4ff,stroke:#9bb8f0,color:#1c3a6e;
The RuntimeManifest is the contract: php_ini directives to preserve, PHP constants and server_vars (with a {fs-root} placeholder resolved at apply time), declarative routes, paths_to_remove (production-only infra stripped after flattening), and extra_directories (dirs outside the WP root, e.g. from auto_prepend_file).
WpcloudHostAnalyzer
Detects WP Cloud / Atomic by the __wp__ directory, WP at doc_root/__wp__/, and the PRIVACY_MODEL env var. Strips the Memcached object-cache.php and wpcomsh mu-plugins, finds auto_prepend_file directories, and declares a thumbnail-generator route — because WP Cloud exports ship only full-size uploads, so missing -WxH sizes are generated on the fly with GD before WordPress even boots.
Plugin deactivation, derived from the manifest
Any paths_to_remove entry under wp-content/plugins/ also triggers removal from the active_plugins option at the end of db-apply — while the DB connection is still open. This avoids "plugin file does not exist" warnings without firing deactivation hooks into code that's already gone from disk. SiteGround strips sg-cachepress and sg-security the same way.
Two route handlers are inlined into the generated runtime.php: the WP Cloud thumbnail generator, and a remote upload proxy that streams missing uploads straight from the source while files-pull is still running — so the site looks complete before the media library finishes downloading.
16End to end — the pull pipeline
Pull (in lib/pull/class-pull.php) is the conductor. It composes the low-level commands into a single resumable pipeline, computing which stages are needed from the options you passed.
flowchart LR P["1 · Preflight"]:::s --> F["2 · Files
(files-pull)"]:::s F --> D["3 · Database
(db-pull)"]:::s D --> A["4 · Import
(db-apply +
URL rewrite)"]:::s A --> FL["5 · Flatten
(flat-docroot)"]:::opt FL --> RT["6 · Runtime
(apply-runtime)"]:::s RT --> ST["7 · Start
(launch server)"]:::s classDef s fill:#2f6df6,stroke:#2455c7,color:#fff; classDef opt fill:#eef4ff,stroke:#9bb8f0,color:#1c3a6e;
db-apply only with a DB target, flat-docroot only with --flatten-to, start only when the runtime is launchable in-process.Two mechanisms make it seamless:
run_until_complete()wraps each network stage in a loop (up to 1,000 iterations): if a stage returns partial (exit 2), it resets status and retries — turning the per-request resume protocol into a single smooth stage from the user's point of view.- Resume + delta —
Pullreads the last completed stage fromstate.pull.stageand skips ahead. If the previous pull already completed,prepare_repull()sets up a delta: it keeps the local index and preflight, clears the transfer state, and deletes the dump + remote index so they're re-fetched. Re-runningpullafter success just syncs what changed.
17State files reference
All state lives as files in --state-dir, written atomically. Progress is computed client-side by reading them — keeping the protocol minimal. The two an integrator cares about are .import-state.json (the brain) and .import-status.json (pollable progress).
| File | Role |
|---|---|
.import-state.json | The canonical resumable state: active command, status, cursor, stage, cached preflight, AIMD tuning, per-phase bookmarks. Corrupt → renamed aside, start fresh. |
.import-status.json | Flat, pollable progress for a UI: step, steps, command, status, phase, error, ts, plus files_done / files_total during fetch. |
.import-index.jsonl | Sorted local file index — the basis for delta detection (what's already on disk). |
.import-remote-index.jsonl | Latest remote tree snapshot from the server. |
.import-download-list.jsonl / -skipped.jsonl | Files the diff decided to fetch / defer (under --filter). |
.import-volatile-files.json | Files that changed mid-stream, with a per-path counter. |
.import-audit.log | Append-only event log (downloads, deletes, volatile detections, errors). Never rotated. |
db.sql / .sql-buffer | The downloaded dump / the in-flight buffer for --sql-output=mysql crash recovery. |
18Distribution & the Playground UI
Two artifacts ship, plus a browser-based wizard that runs the importer entirely client-side.
reprint.phar
Built by Box (bin/build-reprint-phar.sh + box.json). The version comes from git tags into a baked-in VERSION file. The bundled sqlite-database-integration submodule is compiled in, so the phar is self-contained and needs no MySQL on the target. A single CLI guard keyed on IMPORTER_PHAR_ENTRY / IMPORTER_WRAPPER_ENTRY / IMPORTER_WEB_ENTRY lets the same file run as a phar, a source wrapper, or an embedded include.
reprint-exporter-wp.zip
The WordPress plugin, with the exporter Composer package vendored in. stamp-plugin-version.sh stamps the plugin version into index.php + lib.php. The admin UI (Site_Export_Plugin) manages the HMAC secret via a WP option exposed over REST, so platforms can provision it programmatically.
The Playground wizard (reprint-ui/)
A WordPress.com → Playground handoff that runs the migration in the browser and never touches site bytes server-side. It's a stateless PHP app (signed, encrypted cookies — no sessions): the user authorizes via WP.com OAuth, the server enables the exporter and rotates a 60-minute HMAC secret on the chosen Atomic site, then hands off to playground.wordpress.net with a generated Blueprint. Inside Playground, reprint-import.php includes the same reprint.phar and runs pull … --target-engine=sqlite --runtime=none — the browser is the network transport, Playground is the server, and a uploads-proxy mu-plugin streams missing media from the source until the sync completes.
19Key takeaways
Resumption is the spine
Cursors, exit code 2, budgets, atomic state, truncation-on-resume — every layer assumes any request can stop at any moment.
One narrow interface
Everything goes through one HMAC-signed multipart endpoint on a WordPress plugin — the lowest common denominator that works on locked-down hosts.
Pluggable seams
PDO adapters, host analyzers ↔ runtime appliers (via a data manifest), and naive ↔ fast query streams are all swappable behind stable contracts.
Adapts to the host
AIMD tuning and resource budgeting mean the same binary runs well on a 256 MB shared host and a beefy server without configuration.
Structure-preserving rewrites
URL rewriting walks serialized PHP / JSON / block markup without unserialize(), recomputing length prefixes — correctness over shortcuts.
Multiple targets, one path
MySQL, SQLite, nginx, PHP built-in server, and browser Playground all reuse the same import engine.
packages/reprint-exporter/src/class-http-server.php (the router) and export.php (endpoints) for the server; packages/reprint-importer/src/lib/pull/class-pull.php for the orchestration, then ImportClient in import.php for the engine. The README.md documents the CLI contract and state-file formats in full.