Architectural Outline

[tellyworth]

Brief outline and architectural ideas

This repo is intended as a PoC and possible MVP of a sort of "universal importer" for WordPress.

It is built as a WordPress plugin, but otherwise is intended to be written in such a way that it could be used in several contexts such as:

• As an importer plugin on a destination WordPress site
• As an importer on a temporary Playground site as a mid-point for migration (Old Site -> Importer in Playground -> New Site)
• As a hosted migration tool used by web hosts or SaaS migration services

The idea is to have a common "universal" framework for migrating site-to-site. Individual modules would support various types of source input handlers (HTML, RSS, WXR, various export formats); and various types of output handlers (pages, posts, blocks, media files, products, navigation, and so on).

This is all very rough and I expect it will change rapidly as I start experimenting with code and test cases.

Main Concepts

Some of the main pieces I see in the architecture are:

• Site Identifier. A module that can take a web site address, fetch it, and return a bunch of metadata about the site (canonical url, platform, generator, sitemap links, etc).
• Site Indexer. A module that takes a Site Identifier, and produces a list of all of the pages, media, scripts and other resources (perhaps by a combination of crawling and using site maps, RSS feeds and other channels if available)
• Fetcher. Modules that would use http/s (and perhaps other methods like REST APIs, SQL, or FTP) to fetch content from a source site.
• Import Transforms. Modules that would take content received by a Fetcher, and transform it into a simplified interim DOM object.
• Traverser. A module that will traverse interim DOM objects, and identify objects within it (text content, images, comments, products, forms..)
• Output Transforms. Modules that would take objects found by the Traverser, and convert it into WordPress objects like posts, Gutenberg blocks, media files, navigation blocks, e-commerce product items, comments, and so on.

This could well be done on top of the existing WP_Importer class. The idea is to do a similar job, but incorporate transformation steps at both the input and output stages.

PoC/MVP Goal

The minimum viable pieces for something usable would be:

• 1 Site Identifier.
• 1 Site Indexer (sitemap.xml?)
• 1 Fetcher (HTTPS).
• 1 Import transform (HTML, being a useful last-chance fallback for migrating any web site)
• 1 Traverser
• Output Transforms for WP pages, simple Gutenberg blocks, and (maybe) simple navigation.
• (Maybe) some automated tests (most likely integration/e2e).

Expectations

This is experimental code and ideas. It is not intended to conform to the same coding standards as finished production code, WP core, etc.

Once the initial experiments have settled down into something more stable, it could be moved to a WordPress repo.