Fetching Remote Data

Overview #

If you need to fetch data from another server, you should remember that doing so is a relatively slow process and that you can run into problems if there are any timeouts.

To help you to efficiently and robustly fetch your data, we have created two helper functions that you can use.

Both functions are defined in our vip-helper.php file, which can be loaded by including vip-init.php from within your theme’s functions.php file as described in our Development Environment document.

↑ Top ↑

wpcom_vip_file_get_contents() #

wpcom_vip_file_get_contents() works much like PHP’s built-in file_get_contents() function (although it no longer internally uses it). It returns either the HTML result as a string or false on failure. However, it caches and even returns previously cached data if a new remote request fails. We strongly recommend using this function for any remote request that does not require receiving fresh, up-to-the-second results, i.e. anything on the front end of your blog.

  1. The URL you want to fetch. This is the only required argument.
  2. The timeout limit in seconds. Can be 1 to 10 seconds and it defaults to 3 seconds. We strongly discourage using a timeout greater than 3 seconds since remote requests require that the user wait for them to complete before the rest of the page will load.
  3. The minimum cache time in seconds. It cannot be less than 60 and it defaults to 900 (15 minutes). Setting this higher will result in a faster site as remote requests are relatively slow. Results may be cached even longer if the remote sever sends a cache-control header along with its response, and if that value is larger than this value. See below for details and how to disable this.
  4. An array of additional advanced arguments. See below.

The fourth parameter is an optional argument that can be used to set advanced configuration options. The current additional advanced arguments are:

  • obey_cache_control_header — By default if the remote server sends a cache-control header with a max-age value that is larger than the cache time passed as the third parameter of this function, then this remotely provided value will be used instead. This is because it’s assumed that it’s safe to cache data for a longer period of time if the remote server says the data is not going to change. If you wish to ignore the remote server’s header response and forcibly cache for only the time specified by the third parameter, than a function call along these lines should be used:
    echo wpcom_vip_file_get_contents( 'http://example.com/file.txt', 3, 900,
    array( 'obey_cache_control_header' => false ) );
  • http_api_args — Allows you to pass arguments directly to the wp_remote_get() call. See the WordPress.org Code Reference for a list of available arguments. Using this argument will allow you to send things like custom headers or cookies. Example usage:
    echo wpcom_vip_file_get_contents( 'http://example.com/file.txt', 3, 900,
    array( 'http_api_args' => array( 'headers' => array( 'Accept-Encoding' => 'gzip' ) ) ) );

Note that like PHP’s file_get_contents() function, wpcom_vip_file_get_contents() will return the result. You will need to echo it if you want it outputted. This is different from our previous and now deprecated functions, including vip_wp_file_get_contents().

↑ Top ↑

vip_safe_wp_remote_get() #

vip_safe_wp_remote_get() is a sophisticated extended version of wp_remote_get(). It is designed to more gracefully handle failure than wp_safe_remote_get() does. Note that like wp_remote_get() and wp_safe_remote_get, it does not cache. Its arguments are as follows:

  1. The URL you want to fetch. This is the only required argument.
  2. This argument is optional. Pass false if you need to set any of the next arguments.
  3. The number of fails required before subsequent requests automatically return the fallback value. This prevents continually making requests and receiving timeouts for a down or slow remote site. Defaults to 3 retries. Cannot be more than 10.
  4. Number of seconds before the request times out. Can be 1, 2, or 3 and it defaults to 1 second.
  5. This argument controls both the number of seconds before resetting the fail counter and the number of seconds to delay making new requests after the fail threshold is reached. Defaults to 20 and cannot be less than 10.

If you’re confused, here’s some examples that should help clarify:

// Get a URL with a 1 second timeout and cancel remote calls for
// 20 seconds after 3 failed attempts in 20 seconds have occurred
$response = vip_safe_wp_remote_get( $url );
if ( is_wp_error( $response ) )
	echo 'No data is available.';
	echo wp_remote_retrieve_body( $response );

// Get a URL with 1 second timeout and cancel remote calls for 60 seconds
// after 1 failed attempt in 60 seconds has occurred. On failure, display "N/A".
$response = vip_safe_wp_remote_get( $url, false, 1, 1, 60 );
if ( is_wp_error( $response ) )
	echo 'N/A';
	echo wp_remote_retrieve_body( $response );

↑ Top ↑

fetch_feed() #

WordPress’s built-in fetch_feed() function should be used for fetching and parsing feeds. It has built-in caching that defaults to 43200 seconds (12 hours). To change that value, use a filter:

function someprefix_return_900() {
	return 900;

add_filter( 'wp_feed_cache_transient_lifetime', 'someprefix_return_900' );
$feed = fetch_feed( $feed_url );
remove_filter( 'wp_feed_cache_transient_lifetime', 'someprefix_return_900' );

↑ Top ↑

wpcom_vip_wp_oembed_get() #

`wpcom_vip_wp_oembed_get()` is a wrapper for WordPress’ own `wp_oembed_get()` but with added caching.

↑ Top ↑

Uncached Remote Requests #

If for some reason you need to make an uncached remote request, such as to ping an external service during post publish, then you should use the powerful and flexible WordPress HTTP API rather than directly using cURL or some other method like that.

Note that uncached remote requests should never run on the front end of your site for speed and performance reasons.

cURL fopen fsockopen

Ready to get started?

Tell us about your needs

Let us lead the way. We’ll help you select a top tier development partner. We’ll train your developers, operations, infrastructure, and editorial teams. We’ll coarchitect your deployment processes. We will provide live support for peak events. We’ll help your people avoid dark alleys and blind corners, and reduce wasted cycles.