Copying production data to your non-production environments on VIP Go

VIP Go platform specific

This document is for sites running on VIP Go.

Learn more

Overview #

VIP Go has built-in support for syncing/copying data from your production to non-production environments.

This feature facilitates testing and QA of new features and allows teams to accurately reproduce and examine errors in a non-production environment. Our sync process seamlessly and speedily handles the data copying, modifies URLs to match the new environment, and adds a hook for further client-specific customization.

If you have any questions about the data sync process, please contact our Support team.

↑ Top ↑

Running a data sync via CLI #

You will need the VIP CLI installed on your local machine: VIP CLI installation instructions.

Step 1: Find the application name or ID you wish to sync

You can do this with the vip app list command, which will give you back a list of apps. You want the name or the id.

Step 2: Initiate the sync

Run the sync command with the app name, e.g. vip sync --app=my-app, or the app id, e.g. vip sync --app=000. You will be prompted for the environment you wish to sync to, i.e. the target environment. You can get more information about sync command options by using the help parameter, i.e. vip sync --help.

↑ Top ↑

What happens during a Data Sync? #

The operation is designed to not impact the production site in any way; all processing happens on the target environment:

  1. The target environment is protected via “maintenance mode”, this prevents access to the site while the data is being restored and manipulated, and ensures no changes can be made (and lost) during this time
  2. The most recent backup from production is loaded into the target environment
  3. We update all references to production domains to match the target environment
    • The primary production domain is replaced with the primary domain for the target environment; e.g. if your production site is at example.com and your target non-production site is example-com-develop.go-vip.co, then we’ll replace all instances of example.com with example-com-develop.go-vip.co
    • For any additional domains, we use config file with a URL map stored in your site’s code repository. See “domain mapping” below for more details
  4. Caches are flushed
  5. Jetpack is re-connected
  6. Maintenance mode is deactivated
  7. A cron event (vip_go_migration_cleanup) is fired for further cleanup
    • See below for more detail on this hook, and associated considerations

↑ Top ↑

Domain Mapping Config #

If your application has multiple domains in use (this is common in a multisite install, for example), Data Sync needs a bit more information to correctly update URL references in your non-production environments. This can be done using config files in your code repository.

If you don’t have multiple domains, you can skip this section.

Structure #

Data Sync relies on YAML-based config files to correctly handle URL replacements. Here’s what the basic structure looks like:

data_sync:
  domain_map:
    prod-url-1: nonprod-url-1
    prod-url-2: nonprod-url-2

To illustrate, here’s an example. Let’s say your production environment uses the following domains:

  1. en.example.com
  2. fr.example.com

And your non-production environment has the following matching domains:

  1. en-preprod.example.com
  2. fr-preprod.example.com

The config file would look something like this:

data_sync:
  domain_map:
    en.example.com: en-preprod.example.com
    fr.example.com: fr-preprod.example.com

When Data Sync runs, all instances of en.example.com will be replaced with en-preprod.example.com and all instances of fr.example.com will be replaced with fr-preprod.example.com.

If your environments use a mixture of domain-, subdomain- and/or subdirectory-based sites, you can still specify those via config.

Let’s say we had another site in our example above (example.com/translate) which had a matching non-production site (preprod.example.com/translate) our config would look something like this:

data_sync:
domain_map:
en.example.com: en-preprod.example.com
fr.example.com: fr-preprod.example.com
example.com/translate: preprod.example.com/translate

↑ Top ↑

Naming #

If you have multiple applications or environments using the same repo, chances are that each of them will need a unique config. We use a specific naming convention for config files to correctly identify the application and environment:

.vip.[app].[environment].yml

The placeholders are:

  • [app] is the name/slug used to identify your (production) application.
  • [environment] is the environment that this config file applies to.

Here’s an example: .vip.example.develop.yml (this config file would be used for the example application’s develop environment).

In some (very rare) circumstances, you may have multiple environments using the same environment name (e.g. multiple develop environments). In this case, you can add the unique name of your non-production environment at the end (i.e. .vip.[app].[environment].[child].yml; an example would be .vip.example.develop.instance-01.yml).

If you’re not sure what your application or environment name is, please let us know and we’ll be happy to help.

↑ Top ↑

Location #

Config files for Data Sync should live in a config folder in the root of the branch that tracks your production environment (usually master).

Here’s what the root of your master branch might look like:

|– client-mu-plugins/
|– config/
|—– .vip.example.develop.yml
|—– .vip.example.preprod.yml
|– images/
|– languages/
|– plugins/
|– themes/
|– vip-config/

↑ Top ↑

Maintenance #

Please note that you will need to update the config files if any domains are added, removed, or modified across any of your environments. We’ll be adding tools in the future to help simplify this process.

↑ Top ↑

Custom cleanup operations #

The vip_go_migration_cleanup hook can be used for custom cleanup of your application data. Some examples of cleanup operation include removing production API keys or performing brief data manipulation.

Note: The cleanup hook runs after maintenance mode is lifted.

Here’s some example code that shows you how to use this cleanup event:

/**
 * Run some custom cleanup after a migration.
 *
 * @uses vip_go_migration_cleanup
 */
function my_action_vip_go_migration_cleanup() {
	// Safety first: Don't do anything in 
	// the production environment
	if ( 'production' === VIP_GO_ENV ) {
		return;
	}

	delete_option( 'my_social_api_token' );
}
add_action( 'vip_go_migration_cleanup', 'my_action_vip_go_migration_cleanup' );

↑ Top ↑

Media library files and UnionFS #

Instead of copying all the media library files from the production environment, for large media libraries of millions of files and hundreds of gigabytes of data could take many hours, VIP Go provides a service called UnionFS.

UnionFS works by making the files for the production site available to all non-production sites in read-only mode. Files shared by UnionFS in this way are served from the same infrastructure and have the same caching rules applied.

The non-production environment can upload a file of the same name to the same URL, and this will override the file shown on the non-production site (the production file remains untouched).

Caveats:

  • The non-production environment cannot delete a file on the production site. The operation will not error, but the production file will still be shared and available from the non-production environment.
  • If the file is deleted from production, it will no longer be available to be shared with the non-production environments.

Ready to get started?

Drop us a note.

No matter where you are in the planning process, we’re happy to help, and we’re actual humans here on the other side of the form. 👋 We’re here to discuss your challenges and plans, evaluate your existing resources or a potential partner, or even make some initial recommendations. And, of course, we’re here to help any time you’re in the market for some robust WordPress awesomeness.