How to Test and Monitor Azure Blob or Amazon S3 Performance

Exoprise recently released new storage sensors for end-to-end monitoring and testing of Azure Blob or Amazon S3. These sensors, once deployed against a container, enable continuous monitoring of access performance, uptime and availability of Azure Blob storage or Amazon S3. They enable network and storage administrators to test network capacity, latency and effective bandwidth to the various object store regions and datacenters that support the stores.

With both Azure Blob and Amazon S3 sensors, administrators can proactively evaluate cloud object storage solutions and model access from any location or branch office. Many organizations utilize NAS / SAN solutions with cloud-based object storage as a backing store for backup and distribution. Verifying the end-to-end efficacy and performance of these solutions is critical to a healthy storage infrastructure deployment before, during and after a migration.

Cloud Object Stores Overview

Azure Blob and Amazon S3 storage services provide REST and SDK APIs for storing and accessing data in the cloud. Cloud object storage is ideally designed for storing unstructured data such as files and documents and for efficiently distributing access to the files. Common characteristics of the cloud storage solutions include:

• File systems in the cloud with multiple levels of hierarchy.
• Enabling the storage of large files and data reliably and cheaply.
• Advanced protection of content from unauthorized access with a rich set of access control mechanisms.
• Configurable versioning of objects and files.
• File distribution through content delivery networks (CDN) for lower latency and distributed content caching.
• Microsoft Azure recently added an Amazon S3 API Proxy so any application designed to work with S3 will automatically work with Azure Blob stores.

More information about Azure Blob Storage can be found here: https://azure.microsoft.com/en-us/services/storage/blobs/

And more information about Amazon Web Service Simple Scalable Storage (S3) can be found here: https://aws.amazon.com/s3/

Modeling & Testing Performance End-to-End

Easily configure S3 and Azure Blob sensors to test and compare access, uptime and performance of each storage platform from your own networks. The steps are quite simple:

1. Deploy a CloudReady Private Site or choose to run them from our public cloud locations.
   You’ll want to deploy a private site to measure end-to-end performance from the perspective of your network.
2. Create or allocate an Azure Storage Account or an Amazon Access Key for controlling access to a container
3. Create a bucket or container within each system
4. Deploy a sensor with the credentials to the container

CloudReady storage senors will leverage native high-performance APIs (on top of REST) to access the storage from public or private locations. Currently, the sensors perform the following steps as a record of their performance:

• List containers that are available to the account
• List objects that are in the bucket or container
• Upload a random file that is specific for each sensor to the container. High performance uploads are performed to measure end-to-end bandwidth and latency
• Downloads the uploaded file
• Deletes the uploaded file to ensure that the container is left as it originally started. If you are using container versioning then you should be aware that the files will be versioned repeatedly for each sensor run.

High-Level Actions & Low-Level Metrics

With each set of actions against the cloud storage systems, timings are captured for base-lining and comparison. Additionally, low-level metrics are captured that assist with real-time diagnostics of the underlying network and end-to-end infrastructure.

• Upload and download Mbps (Megabits Per Second) for the file uploads and downloads
• List containers time
• List objects time
• Delete object time
• Aggregate DNS Lookup performance that indicate the health of the internal and external name resolution services
• SSL Negotiate time is captured for the transactions and sensor run. Negotiate time is a good indicator of end-to-end network latency
• Aggregate Time to First Byte (TTFB) which indicate the health and response of the Azure Blob and Amazon S3 servers
• End-to-end, hop-by-hop performance is captured giving an instant network trace between the sensor deployment and the cloud object storage service.

Additionally, you get the full set of features that come with CloudReady for monitoring all of your cloud services.

How To Test Amazon S3 Performance

To setup a CloudReady S3 sensor, you’ll need an Access Key attached to an existing account or create a new account, in AWS IAM, and then create an Access Key. If you create a new identity, you can manage and revoke access as you wish. Here are some high-level steps to follow along:

1. Sign into the Amazon Web Services Portal through your IAMs account
2. Depending on if you are using IAMs or not, access your security credentials for the account that you would like to use.
3. Ensure the account has S3 access to a bucket. The S3 sensor lists all buckets available to the account so it will require s3:ListBucket access as well as puts and deletes to the bucket you want to test with.
4. Lastly, ensure the account has access to at least one S3 bucket and that it read, write and delete objects (files) from the bucket. This is what is tested using the S3 sensor.

With these minimal configuration items, you’re ready to configure an Amazon S3 Sensor. Have a look at the following screenshots of the wizard-driven setup.

How To Test Azure Blob Performance

Setting up an Azure Blob sensor is similar to setting up an Amazon S3 sensor. You’ll need to create or use an existing storage account. Here’s a link for more information from Microsoft about creating and using a storage account and steps to follow:

1. Sign into the Azure portal
2. Go to Storage Accounts, click Add to create one
3. Select a subscription for the storage account, enter a name, leave most of the default selections and click Create
4. Next create a blob container or use an existing one. We recommend using an existing one as the sensor will test various list operations, for performance and uptime, and will safely only manipulate files that it creates. Remember the name of the container, you’ll need it to configure the sensor.
5. Finally, create or regenerate an Access key for the account if it doesn’t have one already. You’ll need the access key to configure the sensor.

Here’s the variation on the Azure Blob setup screen.

Amazon S3 & Azure Blob Bandwidth, Performance, & Outage Metrics

Good Reasons to Test and Monitor Cloud Object Stores

There are a number of good reasons to test the end-to-end performance of cloud object storage solutions before during and after migration:

• Network paths to each cloud provider may vary depending on where the data is accessed from.
• Uptime and availability, end-to-end, of the stores is critical to your selection criteria.
• If you are migrating your data storage to a SAN or NAS that leverages a cloud object store behind it, then you will want to ensure you test the solution end-to-end from each location within your organization.
• Its easy to setup and test these solutions. Valuable insight can be had even during a free trial.