Copying production data to your non-production environments on VIP Go

VIP Go platform specific

This document is for sites running on VIP Go.

Learn more

Overview #

VIP Go has built-in support for syncing/copying data from your production to non-production environments.

This feature facilitates testing and QA of new features and allows teams to accurately reproduce and examine errors in a non-production environment. Our sync process seamlessly and speedily handles the data copying, modifies URLs to match the new environment, and adds a hook for further client-specific customization.

To initiate a Data Sync for your environments, please contact our Support team (in the next few months we will be releasing self-service tools for Data Syncs).

↑ Top ↑

What happens during a Data Sync? #

The operation is designed to not impact the production site in any way; all processing happens on the target environment:

  1. The target environment is protected via “maintenance mode”, this prevents access to the site while the data is being restored and manipulated, and ensures no changes can be made (and lost) during this time
  2. A backup from production is loaded into the target environment
    • By default, we’ll use the latest backup, but if you want an earlier backup, please let us know)
  3. We update all references to production domains to match the target environment
    • The primary production domain is replaced with the primary domain for the target environment; e.g. if your production site is at example.com and your target non-production site is example-com-develop.go-vip.co, then we’ll replace all instances of example.com with example-com-develop.go-vip.co
    • For any additional domains, we use config file with a URL map stored in your site’s code repository. See “domain mapping” below for more details
  4. Caches are flushed
  5. Jetpack is re-connected
  6. Maintenance mode is deactivated
  7. A cron event (vip_go_migration_cleanup) is fired for further cleanup
    • See below for more detail on this hook, and associated considerations

↑ Top ↑

Domain Mapping #

A VIP Go environment can have many domains mapped to it, this is especially likely in the case of WordPress multisite installs. For example:

  • A production environment has the primary domain of example.com, and two other domains en-au.example.com, and en-us.example.com.
  • The preprod environment has a primary domain of example-com-preprod.go-vip.co, with two other domains mapped at en-au-example-com-preprod.go-vip.co, and en-us-example-com-preprod.go-vip.co.

Without instruction, the Data Sync process does not know what domains from the production site to replace with what domains from the non-production site.

VIP Go environment config files #

To ensure that the Data Sync process knows to replace en-au.example.com with en-au-example-com-preprod.go-vip.co and  en-us.example.com with en-us-example-com-preprod.go-vip.co, we provide a config file to specify the mapping of domains between environments.

Here’s an example config file, named .vip.example.preprod.yml, which applies to the preprod environment for the example.com WordPress multisite:

data_sync:
  domain_map:
    en-au.example.com: en-au-example-com-preprod.go-vip.co
    en-us.example.com: en-us-example-com-preprod.go-vip.co

↑ Top ↑

Creating and editing config files #

Config files are only used for Data Sync at the moment, and are only required for the target environment, to specify the domain replacements there; so you do not need a config file for your production environment because production is never the target of a sync, syncs always take place moving data from production to a non-production environment.

We have pre-seeded all GitHub repositories with config files for each environment. If you have any questions, or you have a new site which needs a file creating for it, please contact us.

Once the config file is there, you will need to maintain it as any new domains get added to your site.

↑ Top ↑

Which config file applies to which site? #

All config files should be placed in the branch that your production site uses to deploy from, in most cases this will be either master.

Your config file needs to be named in the following pattern: .vip.[app-name].[environment-name].yml, e.g. .vip.example.develop.yml.

In some (currently very rare) circumstances, you may have multiple environments using the same environment name, e.g. multiple develop environments. In this case you add the child site name at the end, e.g .vip.example.develop.instance-01.yml.

↑ Top ↑

Advanced domain mapping in config files #

It’s possible to use the config file to specify that site within a WordPress multisite application uses a mapped domain in production and a domain with a path (e.g. a subdirectory) in the non-production environment; e.g. en-au.example.com in production could be mapped to example-com-preprod.go-vip.co/en-au in the preprod environment, here’s the config for that scenario:

data_sync:
  domain_map:
    en-au.example.com: example-com-preprod.go-vip.co/en-au

↑ Top ↑

Custom cleanup operations #

The vip_go_migration_cleanup hook can be used for custom cleanup of your application data. Some examples of cleanup operation include removing production API keys or performing brief data manipulation.

Note: The cleanup hook runs after maintenance mode is lifted.

Here’s some example code that shows you how to use this cleanup event:

/**
 * Run some custom cleanup after a migration.
 *
 * @uses vip_go_migration_cleanup
 */
function my_action_vip_go_migration_cleanup() {
	// Safety first: Don't do anything in 
	// the production environment
	if ( 'production' === VIP_GO_ENV ) {
		return;
	}

	delete_option( 'my_social_api_token' );
}
add_action( 'vip_go_migration_cleanup', 'my_action_vip_go_migration_cleanup' );

↑ Top ↑

Media library files and UnionFS #

Instead of copying all the media library files from the production environment, for large media libraries of millions of files and hundreds of gigabytes of data could take many hours, VIP Go provides a service we call UnionFS.

UnionFS works by making the files for the production site available to all non-production sites in read-only mode. Files shared by UnionFS in this way are served from the same infrastructure and have the same caching rules applied.

The non-production environment can upload a file of the same name to the same URL, and this will override the file shown on the non-production site (the production file remains untouched).

Caveats:

  • The non-production environment cannot delete a file on the production site. The operation will not error, but the production file will still be shared and available from the non-production environment.
  • If the file is deleted from production, it will no longer be available to be shared to the non-production environments.

Ready to get started?

Drop us a note.

No matter where you are in the planning process, we’re happy to help, and we’re actual humans here on the other side of the form. 👋 We’re here to discuss your challenges and plans, evaluate your existing resources or a potential partner, or even make some initial recommendations. And, of course, we’re here to help any time you’re in the market for some robust WordPress awesomeness.