Using New Relic to Troubleshoot Magento Performance Problems

We are pleased to announce that New Relic Application Performance Monitoring (APM) is now available on our hosting platform! New Relic is already installed on our shared hosting servers and can be installed on our dedicated servers/cloud servers upon request.

New Relic is an application performance and monitoring suite that makes it easy to diagnose and debug performance issues on your website. With New Relic you get valuable insight into your website health. Real-time web transaction times and historical data makes it easy to monitor how your website performs, allowing you to take actions quickly if a problem arise. You can even set up alerts to notify you immediately at certain thresholds.

Some of the features you get with New Relic:

  • Web transactions time: with a quick glance see how well your website performs. New Relic not only monitors home page load time, but all areas of your website including critical functions such as a checkout process or email opt-in form.
  • Transaction throughput: is your server able to keep up with the traffic? Watch your throughput carefully and upgrade before it becomes a problem.
  • Slow MySQL queries: not only will New Relic give you the page response time. It will also show you exactly how much time is spent waiting for a SQL query to finish or waiting for an external web service to respond. This information is invaluable when it comes to optimizing your Magento codebase.
  • Application code profiling: use New Relic to drill down into specific code segments or SQL queries to identify issues that normally are very difficult to find. New Relic will pinpoint the exact line of code that is causing you headache.
  • Key transactions: Flag your most critical transactions to quickly spot when things like response times, call counts, or error rates perform poorly. For a Magento site this would typically be the Add to Cart function or checkout.
  • Apdex user satisfaction scoring: a simplified SLA solution, Apdex measures the ratio of satisfactory response times to unsatisfactory response times and provides a score that gives you better insight into how satisfied users are.

In this blog post we will show of how we can use New Relic to identify and fix a performance problem within a Magento ecommerce site, and also how it can be used to track the performance of your website over time by the help of key transactions tracking. Some of the features mentioned in this article requires New Relic Pro edition. New Relic is available at different subscription levels. To get started, you can sign up for a 14-day trial.

Enable New Relic integration

The first step we need to do is to instruct our website to report data to New Relic. If you are on a ProperHost Magento hosting server we already have the New Relic PHP extension installed, so all you need to do is activate it in .htaccess:

php_value newrelic.enabled yes
php_value newrelic.license "INSERT_YOUR_LICENSE_HERE"
php_value newrelic.appname "INSERT_APP_NAME"

You can find the license key from your New Relic control panel (APM > New PHP Application > Reveal License Key or under Account Settings).

The app name is an arbitrary label which is used to identify the site within the APM interface if you host multiple websites.

After adding the above, New Relic should start collecting metrics and data will show up in your account after a few minutes.

Finding the performance bottleneck

Scenario: customers of our fictitious Magento store have recently reported problems of slow loading pages and not being able to complete their orders. We have also noticed an increase in abandoned carts from our analytics data. For an ecommerce site it is crucial that the checkout process works flawlessly so we turn to New Relic to diagnose the problem. In a complex system such as Magento, finding the piece of code that is responsible for the performance problem can be a challenging task even for an experienced developer, but fortunately New Relic makes it really simple!

We use the APM (Application Performance Monitoring) feature in New Relic to monitor the response time of our web site. The main overview gives you a few key metrics about how well your application is performing, such as the average page load time, application server response time and application throughput.

Click on the application name to show more detailed performance statistics for this application. We are interested in the poorly performing transactions, so we click on Transactions and sort the list by "Slowest average response time". In terms of web applications, a transaction can be thought of as a single request that is processed on the server. New Relic can monitor other type of transactions as well, such as database queries or external web service calls, but in the scope of this tutorial we focus only on web transactions.

We can see that the transaction /checkout/onepage/saveBilling takes on average more than 13 seconds to complete. This is seriously affecting the customer experience and we need to fix this as soon as possible to avoid losing more sales. Below the Transaction traces you will find the recent transactions matching this url and how long it took to process the request.

Next we click on the link "/checkout/onepage/saveBilling" to bring up the Transaction Trace for that url. As we see almost all of the time is spent in the Mage_Checkout_OnepageController::_checkQuoteExternal component. Now click on Trace Details to reveal the complete call tree. We can see that the function _checkQuoteExternal is called from the _getShippingMethodsHtml function in the OnepageController each time the saveBilling action is invoked. This is causing our checkout page to perform poorly.

As a developer we can now inspect the OnepageController, which is located in the file ./app/code/core/Mage/Checkout/controllers/OnepageController.php. The relevant code segment is highlighted below:

protected function _getShippingMethodsHtml() {
	$this->_checkQuoteExternal();
	$layout = $this->getLayout();
	$update = $layout->getUpdate();
	$update->load('checkout_onepage_shippingmethod');
	$layout->generateXml();
	$layout->generateBlocks();
	$output = $layout->getOutput();
	return $output;
}

private function _checkQuoteExternal() {
	sleep(10);
}

As you can see from the above we have added a sleep function inside the code to simulate a performance bottleneck. In a real-world scenario this could for example have been a remote call to a shipping provider or a slow SQL query responsible for the delay.

Tracking key transactions

In web applications, some web transactions are more important to the business than others, and should be monitored with precision. They include key business events in your application (such as sign ups or purchase confirmation) and/or transactions that are particularly important from a performance perspective (such as search or login).

In New Relic, Key Transactions let you closely monitor these important key business transactions and receive alerts when they are performing poorly.

As store owners we obviously want to make sure customers are able to add products to their cart and complete their purchases, so we identify the following two key transactions:

/checkout/cart/add
/checkout/onepage/saveOrder

Let's go ahead and track these in New Relic. From the APM tab, we click on Transactions and select the above transactions one by one, then click on Track as key transaction in the right pane.

These transactions can now be monitored from the Key transactions menu in the top bar. From this section one can also edit the alert policy to automatically be notified via email or phone if certain conditions are met, such as a dissatisfying Apdex score or an increasing server error rate. This way you become aware of any issues before your users notice them.

New Relic Reporting from Magento

For an even deeper integration between Magento and New Relic you can enable New Relic reporting in Magento, and have your store send data and events back to New Relic. For example, the Deployments feature in New Relic is useful to reveal what impact certain code changes (a deployment) have on yoursite's performance. You'll quickly see whether the deployment had a positive or negative impact on Apdex, response time, throughput, or errors.

You can also use New Relic Insights to analyze and visualize data in real-time, helping your team make faster and better decisions.


Magento 2 has built-in support for New Relic Reporting so it is only a matter of enabling it in Stores > Configuration > New Relic Reporting. For Magento 1.x a number of extensions are available that provide similar functionality.

Conclusion

In this article we have shown how New Relic can be used to monitor website performance and gain useful insight into your web sites’ health. For developers, New Relic is an invaluable tool that helps to debug application code and pinpoint the root cause of performance problems, even in a production environment. For more information, visit newrelic.com and sign up for a free 14-day trial now.

How to Upgrade Magento 2 CE using System Upgrade

In this tutorial we will go through the steps necessary to upgrade a Magento Community Edition (CE) 2.x installation using the web-based updater application. At the time of writing, the latest stable version is Magento CE 2.1, but the same steps should apply to future versions as well.

Prerequisites

Starting with the 2.1 release, Magento is no longer supporting PHP 5.5, making PHP 5.6 the new minimum requirement. This is a welcome change as PHP 5.5 will reach End-of-Life (EOL) in July 2016, but it may cause problems for some users as not every hosting provider has made the leap to PHP 5.6 yet.

Magento 2.1 supports both PHP 5.6 and PHP 7, and ProperHost users can freely choose between the two versions. In this tutorial we will use PHP 7, which offers the best performance and latest features. We also assume that Magento is installed in the top-level directory (root) of your account, i.e. /home/<username>/public_html where <username> is your cPanel account username. If you have installed Magento in a sub-directory or add-on domain folder, you need to adjust the paths in the commands accordingly.

Step 0: Backup, backup, backup!

Before you do anything, make sure to create a full backup of your files and database. You can use the cPanel Backup Wizard or create the backup using SSH. We do not recommend that you use the built-in backup functionality in Magento as it can be unreliable.

Step 1: Set PHP 7 as the default for your installation

Create an .htaccess file in your home directory (e.g. /home/youruser/.htaccess) with the following content:

AddType application/x-httpd-php7 .php

This will ensure all .php files in your account is processed by PHP 7. If you want to use PHP 5.6 instead, simply replace php7 with php56 in the line above.

Note: it is important that you save this .htaccess file in your home directory, or a parent directory to your Magento installation. If you add it to your Magento .htaccess it will be overwritten by the Magento upgrader and it will revert to the server default PHP version which may differ from the one you prefer.

Step 2: Configure the cron jobs

Many of the admin backend tasks as well as the upgrade system requires that the Magento 2 cron jobs are configured properly. 

Log in to your cPanel and click on Cron Jobs in the Advanced section. Add three cron job commands and configure them to run Once Per Minute:

php7 -f /home/<username>/public_html/bin/magento cron:run | grep -v "Ran jobs by schedule" >> /home/<username>/public_html/var/log/magento.cron.log
php7 -f /home/<username>/public_html/update/cron.php >> /home/<username>/public_html/var/log/update.cron.log
php7 -f /home/<username>/public_html/bin/magento setup:cron:run >> /home/<username>/public_html/var/log/setup.cron.log

Replace <username> with your cPanel account username. Again, if you prefer to use PHP 5.6 you can replace php7 with php56 in the above commands. When the cron jobs are configured correctly it should look like this:

Step 3: Edit composer.json (Sample Data only)

If you do not have the sample data installed you can skip this step.

Depending on which version of the sample data you have installed previously, you might have to manually correct the component dependencies. If the readiness check fails at “Check Component Dependencies”, open the file composer.json in your installation root directory and change the required version from "100.0.*" to "100.1.*" for each of the sample data modules (see below).

Save the file and continue to Step 4.

Step 4: Set up Marketplace Authentication Keys

In order to use the Magento 2 System Upgrade and Component Manager you first need to set up Magento Marketplace authentication. Create an account and log in to Magento Marketplace, then go to Marketplace > My Access Keys.

From there you can generate a new pair of authentication tokens to use to Install, update, or upgrade third-party components; and upgrade the Magento software using the Component Manager and System Upgrade utilities.

Once you have your public and private keys, log in to your Magento Admin Panel and go to System > Web Setup Wizard > System Configuration. Enter your public and private access keys and click on Save.

Step 5: System Upgrade Wizard

You are now ready to start the upgrade process.

Reminder: have you created a full backup of your account? If not, go back to Step 0 and create a backup first.

From your Magento Admin Panel, go to System > Web Setup Wizard > System Upgrade. Select "Version 2.1.0 CE (latest)" in the Magento Core Components dropdown, and choose "Yes" to upgrade Other Components (such as sample data). If you are upgrading the sample data, make sure that you select the latest version as well (100.1.0), then click on Next.

Magento will now perform a readiness check to ensure that your server fulfills all the requirements to complete the upgrade. When the check has completed, click on Next to continue.

Next, you will now be given the options to have Magento backup your files and database. In our example we use an alternative method (see Step 0: Backup) to perform the backup so we remove the checkboxes and skip this step.

Finally, review your selection and click on Upgrade. Pay special attention to the version numbers to make sure you are upgrading to the correct target version.

Now it is time to sit back and have a cup of coffee while you wait for the upgrade to finish. If any errors occur during the process they will be displayed in the console log.

If everything goes well, you should see a success message after a couple of minutes.

Step 6: Flush Magento Cache

The upgrader application should do this automatically for you, but it is always a good habit to flush cache after any significant changes has been made to your store. Go back to the admin panel, System -> Cache Management and click on Flush Magento Cache. Alternatively you can empty the caches via SSH:

rm -rf var/cache/* var/generation/* var/page_cache/*

That's it! Your Magento 2 store should now be running the latest version.

Bonus Step: Enable production mode and deploy static content (optional)

Before you launch your Magento 2 site you should always switch to production mode for better performance. If you are still developing your site you can skip this step.

After you have changed to production mode you also need to deploy static view files as they will no longer be generated on-the-fly. Below are the commands to change Magento mode and deploy static files using the Magento CLI utility:

php7 bin/magento deploy:mode:set production
php7 bin/magento setup:static-content:deploy

This will take a while so please be patient!

Magento EE & CE 2.1 Is Now Available

Magento today announced the general availability of Magento Enterprise Edition 2.1 and Community Edition 2.1. According to the announcement, this release offers many new features to increase both sales and team productivity, while also simplifying PCI compliance. You can now:

  • Increase conversion rates with a streamlined PayPal In-context Checkout experience.
  • Store credit cards with PayPal so returning customers can quickly check out without re-entering their payment information.
  • Simplify PCI compliance while controlling your checkout design using Braintree Hosted Fields.
  • Improve productivity with new Admin search capabilities and redesigned product, category, and CMS pages.
  • Deliver better search results with a new tool to manage search synonyms at a store view, website, or global level.

Learn more about the new features and enhancements in Magento 2.1 in this blog post or check out the release notes.

New System Requirements

With version 2.1, Magento drop support for PHP 5.5. The new system requirements are:

  • PHP 5.6
  • PHP 7.0.2
  • PHP 7.0.6 + up until 7.1
  • MySQL 5.6

We will be rolling out an update to all our M2 hosting servers in the next few days to accommodate the latest requirements.