AWS Snowball

AWS Snowball accelerates moving large amounts of data into and out of AWS using secure Snowball appliances. The Snowball appliance is purpose-built for efficient data storage and transfer. All AWS Regions have 80 TB Snowballs while US Regions have both 50 TB and 80 TB models. The Snowball appliance is rugged enough to withstand an 8.5-G jolt. At less than 50 pounds, the appliance is light enough for one person to carry. It is entirely self-contained, with a power cord, one RJ45 1 GigE and two SFP+ 10 GigE network connections on the back and an E Ink display and control panel on the front. Each Snowball appliance is water-resistant and dustproof and serves as its own rugged shipping container. AWS transfers users data directly onto and off of Snowball storage devices using Amazon’s high-speed internal network and bypasses the Internet.

  • For datasets of significant size, Snowball is often faster than Internet transfer and more cost effective than upgrading connectivity.
  • AWS Snowball supports importing data into and exporting data from Amazon S3 buckets. From there, the data can be copied or moved to other AWS services such as Amazon EBS and Amazon Glacier as desired.
  • Users can perform local data transfers between on-premises data center and a Snowball. By doing these, users can transfers through the Snowball client, a standalone downloadable client. 
  • The Snowball is its own shipping container, and its E Ink display changes to show the shipping label when the Snowball is ready to ship.
  • Users can manage jobs through the AWS Snow Family Management Console or programmatically with the job management API.
AWS Snowball

AWS Snowball Benefits

AWS Snowball Jobs are created in the AWS Management Console. Once a job is created, AWS automatically ships a pre-provisioned Snowball Edge device to users location. When receiving the device, users can simply attach it to the local network and connect the applications. Once the device is ready to be returned, the E Ink shipping label automatically updates, and the freight carrier transports it to the correct AWS facility where the upload begins. Job status can be tracked via Amazon SNS-generated text or email messages or directly in the AWS Management Console.

AWS Snowball run EC2 AMIs and deploy AWS Lambda code on Snowball Edge devices to run local processing or analysis with machine learning or other applications. Users can run applications directly on the device as a consistent AWS environment without network connectivity. This capability helps users develop the machine learning and analysis tools and test them in the cloud, but operate them in locations with limited or non-existent network connections before shipping the data back to AWS. 

 

AWS Snowball Edge devices can provide local storage to existing on-premises applications through a file sharing protocol (NFS) or object storage interface (the S3 API). Additionally, users can use on-board block storage volumes for applications running on Amazon EC2 instances on the Snowball. They can also cluster Snowball Edge devices together into a single, larger storage tier with increased durability. If a Snowball Edge device needs to be replaced, it can be removed from the cluster and replaced with a new device.

AWS Snowball moves terabytes of data in about a week. Users can use it to move things like databases, backups, archives, healthcare records, analytics datasets, IoT sensor data and media content, especially when network conditions prevent realistic timelines for transferring large amounts of data both into and out of AWS. AWS Snowball can transport multiple terabytes of data and multiple devices can be used in parallel or clustered together to transfer petabytes of data into or out of AWS. Snowball is currently available in select regions and users location will be verified once they create a job in the AWS Management Console.

 

AWS Snowball Features

Performance

The Snowball appliance is purpose-built for efficient data storage and transfer, including a high-speed, 10 Gbps network connection designed to minimize data transfer times, allowing you to transfer up to 80 TB of data from customers data source to the appliance in 2.5 days, plus shipping time. In this case, the end-to-end time to transfer the data into AWS is approximately a week, including default shipping and handling time to AWS data centers. Copying 160 TB of data can be completed in the same amount of time by using two 80 TB Snowballs in parallel.

  • Users can use the Snowball client to estimate the time it takes to transfer the data. In general, users can improve the transfer speed from the data source to the Snowball appliance by reducing local network use, eliminating unnecessary hops between the AWS Snowball appliance and the workstation, using a powerful computer as the workstation, and combining smaller objects.
  • Parallelization can also help achieve maximum performance of data transfer. This could involve one or more of the following parallelization types: using multiple instances of the Snowball client on a single workstation with a single Snowball appliance; using multiple instances of the Snowball client on multiple workstations with a single Snowball appliance; and/or using multiple instances of the Snowball client on multiple workstations with multiple Snowball appliances.
Interfaces

There are two ways to get started with Snowball. Users can create an import or export job using the AWS Snowball Management Console or the Snowball Job Management API and integrate AWS Snowball as a part of the data management solution. The primary functions of the API are to create, list, and describe import and export jobs, and it uses a simple standards-based REST web services interface. Users have two ways to locally transfer data between a Snowball appliance and on-premises data center.

  • The Snowball client, available as a download from the AWS Import/Export Tools page, is a standalone terminal application that users run on the local workstation to do the data transfer. Customers can use simple copy (cp) commands to transfer data, and handling errors and logs are written to the local workstation for troubleshooting and auditing.
  • The second option to locally transfer data between a Snowball appliance and on-premises data center is the Amazon S3 Adapter for Snowball, which is also available as a download from the AWS Import/Export Tools page. Users can programmatically transfer data between on-premises data center and a Snowball appliance using a subset of the Amazon S3 REST API commands. This allows users to have direct access to a Snowball appliance as if it were an Amazon S3 endpoint. By default, the adapter runs on port 8080, but a different port can be specified by changing the adapter.config file.
Scalability and Elasticity

Each AWS Snowball appliance is capable of storing 50 TB or 80 TB of data. When transferring more data than that, users can use multiple appliances. For Amazon S3, individual files are loaded as objects and can range up to 5 TB in size, but users can load any number of objects in Amazon S3. The aggregate total amount of data that can be imported is virtually unlimited.

Security

Users can integrate Snowball with IAM to control which actions a user can perform. Customers can give the IAM users on the AWS account access to all Snowball actions or to a subset of them. Similarly, an IAM user that creates a Snowball job must have permissions to access the Amazon S3 buckets that will be used for the import operations.

  • For Snowball, AWS KMS protects the encryption keys used to protect data on each Snowball appliance. All data loaded onto a Snowball appliance is encrypted using 256-bit encryption.
  • AWS Snowball is physically secured by using an industry- standard Trusted Platform Module (TPM) that uses a dedicated processor designed to detect any unauthorized modifications to the hardware, firmware, or software.
  • AWS Snowball is included in the AWS HIPAA compliance program so customers can use Snowball to transfer large amounts of Protected Health Information (PHI) data into and out of AWS.54
Usage Patterns

AWS Snowball is ideal for transferring anywhere from terabytes to many petabytes of data in and out of the AWS Cloud securely. This is especially beneficial in cases where users don’t want to make expensive upgrades to the network infrastructure or in areas where high-speed Internet connections are not available or cost prohibitive. In general, if loading  data over the Internet would take a week or more, users should consider using Snowball.

  • Common use cases include cloud migration, disaster recovery, data center decommission, and content distribution. When decommissioning a data center, many steps are involved to make sure valuable data is not lost, and Snowball can help ensure data is securely and cost-effectively transferred to AWS.
  • In a content distribution scenario, users might use Snowball appliances when regularly receive or need to share large amounts of data with clients, customers, or business associates.
  • AWS Snowball appliances can be sent directly from AWS to client or customer locations. AWS Snowball might not be the ideal solution the data can be transferred over the Internet in less than one week.
  • Snowball Edge devices feature high-speed network connections, supporting 10 Gbps to 100 Gbps links with RJ45, SFP+ and QSFP+ copper, and optical interfaces. All encryption is performed on the device itself, helping enable a higher data throughput rate and shorter data transfer times. 
Built for edge computing & data transfer

AWS Snowball service delivers users Snowball Edge devices with storage and optional Amazon EC2 and AWS IOT Greengrass compute in shippable, hardened, secure cases. With AWS Snowball, users bring cloud capabilities for machine learning, data analytics, processing, and storage to the edge, for migrations, short-term data collection, or even long-term deployments. AWS Snowball devices work with or without the internet, do not require a dedicated IT operator, and are designed to be used in remote environments.

  • With AWS Snowball, users have the choice of two devices, Snowball Edge Compute Optimized with more computing capabilities, suited for higher performance workloads, or Snowball Edge Storage Optimized with more storage, which is suited for large-scale data migrations and capacity-oriented workloads.

Snowball Edge Compute Optimized provides powerful computing resources for use cases such as machine learning, full motion video analysis, analytics, and local computing stacks. These capabilities include 52 vCPUs, 208 GiB of memory, and an optional NVIDIA Tesla V100 GPU. For storage, the device provides 42 TB usable HDD capacity for S3 compatible object storage or EBS-compatible block volumes, as well as 7.68 TB of usable NVMe SSD capacity for EBS-compatible block volumes.

  • Snowball Edge Compute Optimized devices run Amazon EC2 sbe-c and sbe-g instances, which are equivalent to C5, M5a, G3, and P3 instances.

Snowball Edge Storage Optimized devices are well suited for large-scale data migrations and recurring transfer workflows, as well as local computing with higher capacity needs. Snowball Edge Storage Optimized provides 80 TB of HDD capacity for block volumes and Amazon S3-compatible object storage, and 1 TB of SATA SSD for block volumes.

  • For computing resources, the device provides 40 vCPUs, and 80 GiB of memory to support Amazon EC2 sbe1 instances (equivalent to C5).
Rugged and portable

AWS Snowball Edge devices have a ruggedized case designed for durability and portability. A device weighs less than 50 pounds and can be moved by a single person.

Tamper evident

Snowball Edge devices feature a Trusted Platform Module (TPM) that provides a hardware root of trust. The TPM also provides interfaces to the trusted software stack during the measurements and verification of the boot environment integrity after the power is switched on, and before the Snowball Edge device is ready to be used. AWS also uses additional tamper-indicating inspection processes after each device is received back to the AWS Region. This helps to ensure the integrity of the AWS Snowball Edge device, and with the AWS Snowball service’s encryption features, it helps preserve the confidentiality of the data.

AWS OpsHub

AWS OpsHub is a graphical user interface that makes it easy to set up and manage AWS Snowball devices enabling users to rapidly deploy edge computing workloads and simplify data migration to the cloud. Users can download and install AWS OpsHub on any Windows or Mac client machine, such as a laptop.

GPU support

Snowball Edge Compute Optimized provides an optional NVIDIA Tesla V100 GPU along with Amazon EC2 instances to accelerate an application’s performance in disconnected environments. Using the GPU option, users can run applications such as advanced machine learning and full motion video analysis in environments with little or no connectivity.

Clustering

Users can cluster multiple Snowball Edge devices when running edge computing jobs to create a local storage tier with increased durability for on-premises applications. When creating a new job in the Console, select the option to make a cluster. In the event of a device failure, a replacement device can be ordered easily through the Console. This functionality is available for local storage and compute jobs and is not enabled for data transfer jobs.

NFS endpoint

Applications can work with Snowball Edge as a NFS mount point. NFS v3 and v4.1 are supported so users can easily use Snowball Edge with the existing on-premises servers and file-based applications. Using the NFS interface allows simple file transfer to a Snowball Edge device when the S3 adapter is not feasible. The file system metadata is preserved until the files are converted into objects when they are transferred into the S3 bucket.

  • Applications can work with Snowball Edge object storage through an S3-compatible endpoint accessed through the S3 SDK or CLI. 
End-to-end tracking

AWS Snowball devices use an innovative, E Ink shipping label designed to ensure the device is automatically sent to the correct AWS facility and also aids in tracking. Once users have completed your data transfer job, it can be tracked via Amazon SNS, text messages, and via the AWS Console.

Block storage

Users can run block storage on both Snowball Edge Compute Optimized and Snowball Edge Storage Optimized devices. Users attach block storage volumes to Amazon EC2 instances using a subset of the Amazon EBS API that enable users to configure and manage volumes for EC2 instances on Snowball Edge devices. This makes it easier to develop applications in EC2, and then run them in disconnected and remote locations. Snowball Edge supports both performance optimized and capacity optimized volume types.

AWS Snowball Device

The Snowball and the Snowball Edge are two different devices. This guide is for the Snowball. Both devices allow users to move huge amounts of data into and out of Amazon S3, they both have the same job management API, and they both use the same console. However, the two devices differ in hardware specifications, some features, what transfer tools are used, and price. The following outlines the different tools used with the AWS Snowball devices, and how they are used:

Snowball Tools

Snowball client with Snowball

  • Users can download and install the Snowball client from AWS Snowball resources. The Snowball client must be downloaded from the AWS Snowball Resources page and installed on a powerful workstation that users own.
  • Users can transfer data to or from the Snowball, using the Snowball Client.
  • Encrypts data on users powerful workstation before the data is transferred to the Snowball.

Amazon S3 Adapter for Snowball with Snowball

AWS Snowball device
AWS Snowball device
AWS snowball logo
Snowball Edge Tools

Snowball client with Snowball Edge

Amazon S3 Adapter for Snowball with Snowball Edge

  • Is already installed on the Snowball Edge by default. It does not need to be downloaded or installed.
  • Can transfer data to or from the Snowball Edge, using the Amazon S3 Adapter
  • Encrypts data on the Snowball Edge while the data is transferred to the device.

File interface with Snowball Edge

  • Is already installed on the Snowball Edge by default. It does not need to be downloaded or installed.
  • Can transfer data by dragging and dropping files up to 150 GB in size from users computer to the buckets on the Snowball Edge through an easy-to-configure NFS mount point. Users can use the File Interface for the AWS Snowball Edge.
  • Encrypts data on the Snowball Edge while the data is transferred to the device.

AWS IoT Greengrass console with Snowball Edge

  • With a Snowball Edge, you can use the AWS IoT Greengrass console to update your AWS IoT Greengrass group and the core running on the Snowball Edge.

AWS Snowball Edge device
AWS Snowball Edge device

AWS Snowball Jobs

AWS Snowball Concepts

Each import job uses a single Snowball appliance. After you create a job in the AWS Snow Family Management Console or the job management API, AWS ship users a Snowball. When it arrives in a few days, users will connect the Snowball to the network and transfer the data that the user want imported into Amazon S3 onto that Snowball using the Snowball client or the Amazon S3 Adapter for Snowball. When done transferring data, ship the Snowball back to AWS, and AWS will import the user data into Amazon S3.

  • Each export job can use any number of Snowball appliances. After the user create a job in the AWS Snow Family Management Console or the job management API, a listing operation starts in Amazon S3. This listing operation splits the job into parts. Each job part can be up to about 80 TB in size, and each job part has exactly one Snowball associated with it. After the job parts are created, the user first job part enters the Preparing Snowball status.
  • Soon after that, AWS start exporting the data onto a Snowball. Typically, exporting data takes one business day. However, this process can take longer. Once the export is done, AWS gets the Snowball ready for pickup by the region’s carrier. When the Snowball arrives at the data center or office in a few days, the user connect the Snowball to the network and transfer the data that needed to be exported to the servers by using the Snowball client or the Amazon S3 Adapter for Snowball.
Job Details

Each import or export job for Snowball is defined by the details that you specify when it’s created. The following list describes all the details of a job.

  • Job name – A name for the job, containing alphanumeric characters, spaces, and any Unicode special characters.
  • Job type – The type of job, either import or export.
  • Job ID – A unique 39-character label that identifies your job. The job ID appears at the bottom of the shipping label that appears on the E Ink display, and in the name of a job’s manifest file.
  • Created date – The date that you created this job.
  • Shipping speed – Speed options are based on region. 
  • IAM role ARN – This Amazon Resource Name (ARN) is the AWS Identity and Access Management (IAM) role that is created during job creation with write permissions for users Amazon S3 buckets. The creation process is automatic, and the IAM role that users allow Snowball to assume is only used to copy the data between Amazon S3 buckets and the Snowball. 
  • AWS KMS key – In Snowball, AWS Key Management Service (AWS KMS) encrypts the keys on each Snowball. When creating job, users also choose or create an ARN for an AWS KMS encryption key that you own. 
  • Snowball capacity – In the US regions, Snowballs come in two sizes: 50 TB (42 TB usable capacity) and 80 TB (72 TB usable capacity). All other regions have the 80 TB Snowballs only. 
  • Storage service – The AWS storage service associated with this job, in this case Amazon S3. 
  • Resources – The AWS storage service resources associated with users job. In this case, these are the Amazon S3 buckets that the data is transferred to or from.
 

A job in AWS Snowball (Snowball) is a discrete unit of work, defined when users create it in the console or the job management API. Jobs have types, details, and statuses. Each of those elements is covered in greater detail in the sections that follow.

Job Types

There are two different job types: import jobs and export jobs. Both of the Snowball job types are summarized following, including the source of the data, how much data can be moved, and the result users can expect at successful job completion. Although these two types of jobs have fundamental differences, they share some common details The source can be local to users data center or office, or it can be an Amazon S3 bucket.

Import into Amazon S3: An import job is the transfer of 72 TB or less of users data (located in an on-premises data source), copied onto a single Snowball, and then moved into Amazon S3. For import jobs, Snowballs and jobs have a one-to-one relationship, meaning that each job has exactly one Snowball associated with it. When additional Snowballs is needed, users can create new import jobs or clone existing ones.

  • The data source for an import job should be on-premises. In other words, the storage devices that hold the data to be transferred should be physically located at the address that users provided when creating the job.
  • Users can import any number of directories, files, and objects for each import job, provided the amount of data importing fits within a single Snowball. 
  • When importing files, each file becomes an object in Amazon S3 and each directory becomes a prefix. When importing data into an existing bucket, any existing objects with the same names as newly imported objects will be overwritten.
  • When the import has been processed and verified, AWS performs a complete erasure of the Snowball. This erasure follows the National Institute of Standards and Technology (NIST) 800-88 standards.
  • After importing is complete, users can download a job report. This report alerts them to any objects that failed the import process. 
 

Export from Amazon S3: An export job is the transfer of any amount of data (located in Amazon S3), copied onto any number of Snowballs, and then moved one Snowball at a time into on-premises data destination. When creating an export job, it’s split into job parts. Each job part is no more than 72 TB in size, and each job part has exactly one Snowball associated with it.

  • Users data source for an export job is one or more Amazon S3 buckets. Once the data for a job part is moved from Amazon S3 to a Snowball, users can download a job report. This report will alert users to any objects that failed the transfer to the Snowball. 
  • Users can export any number of objects for each export job, using as many Snowballs as it takes to complete the transfer. Snowballs for an export job’s job parts are delivered one after another, with subsequent Snowballs shipping out to users once the previous job part has entered the In transit to AWS status.
  • When copying objects into on-premises data destination from a Snowball, those objects are saved as files. When copying objects into a location that already holds files, any existing files with the same names will be overwritten.
  • When AWS receives a returned Snowball, AWS perform a complete erasure of the Snowball. This erasure follows the NIST 800-88 standards.

AWS Snowball Edge

AWS Snowball

AWS Snowball Edge is a type of Snowball device with on-board storage and compute power for select AWS capabilities. Snowball Edge can do local processing and edge-computing workloads in addition to transferring data between users local environment and the AWS Cloud. Each Snowball Edge device can transport data at speeds faster than the internet. This transport is done by shipping the data in the appliances through a regional carrier. The appliances are rugged, complete with E Ink shipping labels. Snowball Edge devices have three options for device configurations—Storage OptimizedCompute Optimized, and Compute Optimized with GPU. When this guide refers to Snowball Edge devices, it’s referring to all options of the device. When specific information applies only to one or more optional configurations of devices (such as how the Snowball Edge with GPU has an on-board GPU), it is called out specifically. 

  • Large amounts of storage capacity or compute functionality for devices. This depends on the options users choose when creating the job.
  • Network adapters with transfer speeds of up to 100 GB/second. 
  • Encryption is enforced, protecting data at rest and in physical transit. 
  • Users can import or export data between local environments and Amazon S3, and physically transport the data with one or more devices without using the internet.
  • AWS Snowball Edge devices are their own rugged box. The built-in E Ink display changes to show users shipping label when the device is ready to ship.
  • AWS Snowball Edge devices come with an on-board LCD display that can be used to manage network connections and get service status information.
  • Users can cluster AWS Snowball Edge devices for local storage and compute jobs to achieve data durability across 5–10 devices and locally grow or shrink storage on demand.
  • Users can use the file interface to read and write data to an AWS Snowball Edge device through a file share or Network File System (NFS) mount point.
  • Users can write Python-language Lambda functions and associate them with Amazon S3 buckets when creating an AWS Snowball Edge device job. Each function triggers when a local Amazon S3 PUT object action is run on the associated bucket on the device.
  • AWS Snowball Edge devices have Amazon S3 and Amazon EC2 compatible endpoints available, enabling programmatic use cases.
  • AWS Snowball Edge devices support the new sbe1sbe-c, and sbe-g instance types, which you can use to run compute instances on the device using Amazon Machine Images (AMIs).

Device Options

 
 

Snowball Edge devices have the following options for device configurations:

  • Snowball Edge Storage Optimized (for data transfer) – This Snowball Edge device option has a 100 TB (80 TB usable) storage capacity. 
  • Snowball Edge Storage Optimized (with EC2 compute functionality) – This Snowball Edge device option has up to 80 TB of usable storage space, 24 vCPUs, and 32 GiB of memory for compute functionality. It also comes with 1 TB of additional SSD storage space for block volumes attached to Amazon EC2 AMIs.
  • Snowball Edge Compute Optimized – This Snowball Edge device option has the most compute functionality, with 52 vCPUs, 208 GiB of memory, and 42 TB (39.5 usable) plus 7.68 TB of dedicated NVMe SSD for compute instances for block storage volumes for EC2 compute instances, and 42 TB of HDD capacity for either object storage or block storage volumes.
  • Snowball Edge Compute Optimized with GPU – This Snowball Edge device option is identical to the Compute Optimized option, except for an installed GPU, equivalent to the one available in the P3 Amazon EC2 instance type. It has a storage capacity of 42 TB (39.5 TB of HDD storage that can be used for a combination of Amazon S3 compatible object storage and Amazon EBS compatible block storage volumes) plus 7.68 TB of dedicated NVMe SSD for compute instances.
Snowball Edge Tools
AWS OpsHub for Snow Family
  • The Snow Family Devices now offer a user-friendly tool, AWS OpsHub for Snow Family, that users can use to manage the devices and local AWS services. Customers use AWS OpsHub on a client computer to perform tasks such as unlocking and configuring single or clustered devices, transferring files, and launching and managing instances running on Snow Family Devices. 

AWS Snowball client with Snowball Edge
  • Download the Snowball client from the AWS Snowball Edge Resources page and install it on your own computer.
  • Use Snowball client to unlock the Snowball Edge or the cluster of Snowball Edge devices. 
  • The Snowball client doesn’t transfer data.
Amazon S3 Adapter for Snowball with Snowball Edge
  • Is already installed on the Snowball Edge by default. It does not need to be downloaded or installed.
  • Can transfer data to or from the Snowball Edge, using the Amazon S3 Interface.
  • Encrypts data on the Snowball Edge while the data is transferred to the device.
File interface with Snowball Edge
  • Is already installed on the Snowball Edge by default. It does not need to be downloaded or installed.
  • Can transfer data by dragging and dropping files up to 150 GB in size from users computer to the buckets on the Snowball Edge through an easy-to-configure NFS mount point. 
  • Encrypts data on the Snowball Edge while the data is transferred to the device.
AWS IoT Greengrass console with Snowball Edge
  • With a Snowball Edge, you can use the AWS IoT Greengrass console to update your AWS IoT Greengrass group and the core running on the Snowball Edge.
 

AWS Snowball Edge devices are owned by AWS, and they reside at users on-premises location while they’re in use. There are three job types users can use with an AWS Snowball Edge device. Although the job types differ in their use cases, every job type has the same workflow for how the user order, receive, and return devices.

How Import Jobs Work

Each import job uses a single Snowball appliance. After creating a job in the AWS Snow Family Management Console or the job management API, AWS ship a Snowball to the user. When it arrives in a few days, users connect the Snowball Edge device to the network and transfer the data that the user want imported into Amazon S3 onto the device. When the user done transferring data, ship the Snowball back to AWS, and AWS import the data into Amazon S3.

How Export Jobs Work

Each export job can use any number of AWS Snowball Edge devices. If the listing contains more data than can fit on a single device, multiple devices are provided to users. Each job part has exactly one device associated with it. After the job parts are created, the first job part enters the Preparing Snowball status. Soon after that, AWS start exporting the data onto a device. Typically, exporting data takes one business day. However, this process can take longer depending on the amount and type of data. When the export is done, AWS gets the device ready for pickup by the region’s carrier. When it arrives, users connect the AWS Snowball Edge device to the network and transfer the data that the user want to import from Amazon S3 onto the device.

When done transferring data, ship the device back to AWS. When AWS receive the device for the export job part, AWS erase it completely. This erasure follows the National Institute of Standards and Technology (NIST) 800-88 standards. This step marks the completion of that particular job part.

  • For keylisting: Before we export the objects in the S3 bucket, we scan the bucket. If the bucket is altered after the scan, the job could encounter delays because we scan for missing or altered objects.

  • For S3 Glacier: It is important to note that AWS Snowball cannot export objects in the S3 Glacier storage class. These objects must be restored before AWS Snowball can successfully export the objects in the bucket.

How Local Compute and Storage Jobs Work

Users can use the local compute and storage functionality of an AWS Snowball Edge device with all job types in AWS Regions that support Lambda. The compute functionality is named AWS Lambda powered by AWS Greengrass, where Python-language AWS Lambda functions can be triggered by Amazon S3 PUT object actions on buckets specified when the user created the job. 

How a Clustered Local Compute and Storage Job Works

A cluster job is a special kind of job for local storage and compute only. It is for those workloads that require increased data durability and storage capacity. Clusters have 5–10 AWS Snowball Edge devices, called nodes. When users receive the nodes from the regional carrier, connect all the nodes to power and the network to obtain their IP addresses. Users can use these IP addresses to unlock all the nodes of the cluster at once with a single unlock command, using the IP address of one of the nodes. 

  • Users can write data to an unlocked cluster by using the Amazon S3 Adapter for Snowball or the NFS mount point through the leader node and the data distributed among the other nodes.
  • When done with cluster, users need to ship all the nodes back to AWS. When AWS receive the cluster node, AWS perform a complete erasure of the Snowball. This erasure follows the National Institute of Standards and Technology (NIST) 800-88 standards.
  • Like standalone local storage and compute jobs, the data stored in a cluster can’t be imported into Amazon S3 without ordering additional devices as a part of separate import jobs. When ordering these devices, users can transfer the data from the cluster to the devices and import the data when they return the devices for the import jobs.

Snowball Edge Works

 
 

 

Device Specifications

 
 
Supported Network Hardware

To use the AWS Snowball Edge device, users need their own network cables. For RJ45 cables, there are no specific recommendations. SFP28 and QSFP28 cables and modules from Mellanox and Finisar have been verified to be compatible with the device. After opening the back panel of the AWS Snowball Edge device, users see the network ports shown in the following figure.


                The available network ports

These ports support the following network hardware.

SFP: This port provides a 10G/25G SFP28 interface compatible with SFP28 and SFP+ transceiver modules and direct-attach copper (DAC) cables. Users need to provide their own transceivers or DAC cables.

  • For 10G operation, users can use any SFP+ option. Examples include:  

    • 10Gbase-LR (single mode fiber) transceiver

    • 10Gbase-SR (multi-mode fiber) transceiver

    • SFP+ DAC cable

  • For 25G operation, users can use any SFP28 option. Examples include:

    • 25Gbase-LR (single mode fiber) transceiver

    • 25Gbase-SR (multi-mode fiber) transceiver

    • SFP28 DAC cable


                SFP+ Copper

QSFP: This port provides a 40G QSFP28 interface on storage-optimized devices and a 40/50/100G QSFP28 interface on compute-optimized devices. Both are compatible with QSFP+ transceiver modules and DAC cables. Users need to provide their own transceivers or DAC cables. The following are some Examples:

  • 40Gbase-LR4 (single mode fiber) transceiver

  • 40Gbase-SR4 (multi-mode fiber) transceiver

  • QSFP28 DAC


                QSFP+

RJ45: This port provides 1Gbase-TX/10Gbase-TX operation. It is connected via UTP cable terminated with an RJ45 connector. Compute-optimized devices have two RJ45 ports. 

  • 1G operation is indicated by a blinking amber light. 1G operation is not recommended for large-scale data transfers to the Snowball Edge device, as it dramatically increases the time it takes to transfer data.
  • 10G operation is indicated by a blinking green light. It requires a Cat6A UTP cable with a maximum operating distance of 180 feet (55 meters).

                RJ45

cluster is a logical grouping of AWS Snowball Edge devices, in groups of 5–10 devices. A cluster is created as a single job, which offers increased durability and storage capacity. This section provides conceptual, usage, and administrative information about Snowball Edge clusters, in addition to walkthroughs for common Snowball Edge procedures.

Clustering Overview

For the AWS Snowball service, a cluster is a collective of Snowball Edge devices used as a single logical unit for local storage and compute purposes. A cluster offers two primary benefits over a standalone Snowball Edge for local storage and computing:

  • Increased durability – The data stored in a cluster of Snowball Edge devices enjoys increased data durability over a single device. In addition, the data on the cluster remains as safe and viable as it was previously, despite possible Snowball Edge outages in the cluster. Clusters can withstand the loss of two nodes before the data is in danger. Users can also add or replace nodes.
  • Increased storage – The total available storage is 45 terabytes of data per node in the cluster. Thus, in a five-node cluster, there are 225 terabytes of available storage space. In contrast, there are about 80 terabytes of available storage space in a standalone Snowball Edge. Clusters that have more than five nodes have even more storage space.
 

A cluster of Snowball Edge devices is made of leaderless nodes. Any node can write data to and read data from the entire cluster, and all nodes are capable of performing the behind-the-scenes management of the cluster.

Snowball Edge Cluster Quorums

A quorum represents the minimum number of Snowball Edge devices in a cluster that must be communicating with each other to maintain some level of operation. There are two levels of quorum for Snowball Edge clusters—a read/write quorum and a read quorum. Suppose that the user upload the data to a cluster of Snowball Edge devices. With all devices healthy, the user have a read/write quorum for your cluster. If one of those nodes goes offline, the user reduce the operational capacity of the cluster. However, the user can still read and write to the cluster. In that sense, with the cluster operating all but one node, the cluster still has a read/write quorum.

  • If two nodes in the cluster are down, any additional or ongoing write operations fail. But any data that was successfully written to the cluster can be accessed and read. This is called a read quorum.

Finally, suppose that a third node loses power. Then the cluster is offline, and the data in the cluster is unavailable. Users might be able fix this, or the data might be permanently lost, depending on the severity of the event. If it is a temporary external power event, and you can power the three Snowball Edge devices back on and unlock all the nodes in the cluster, your data is available again.

Users can determine the quorum state of the cluster by determining the node’s lock state and network reachability. The snowballEdge describe-cluster command reports back the lock and network reachability state for every node in an unlocked cluster. Ensuring that the devices in the cluster are healthy and connected is an administrative responsibility that users take on when creating the cluster job. 

Considerations for Cluster Jobs for AWS Snowball Edge

Keep the following considerations in mind when planning to use a cluster of Snowball Edges:

  • AWS recommend that users have a redundant power supply to reduce potential performance and stability issues for the cluster. 
  • As with standalone local storage and compute jobs, the data stored in a cluster can’t be imported into Amazon S3 without ordering additional devices as a part of separate import jobs. If users order these devices, they can transfer the data from the cluster to the devices and import the data when returning the devices for the import jobs.
  • To get data onto a cluster from Amazon S3, create a separate export job and copy the data from the devices of the export job onto the cluster. 
  • Users can create a cluster job from the console, the AWS CLI, or one of the AWS SDKs. 
  • Cluster nodes have node IDs. A node ID is the same as the job ID for a device that users can get from the console, the AWS CLI, the AWS SDKs, and the Snowball client. Users can use node IDs to remove old nodes from clusters. Users can get a list of node IDs by using the snowballEdge describe-device command on an unlocked device or the describe-cluster on an unlocked cluster. 
  • The lifespan of a cluster is limited by the security certificate granted to the cluster devices when the cluster is provisioned. By default, Snowball Edge devices can be used for up to 365 days before they need to be returned. At the end of that time, the devices stop responding to read/write requests. 
  • When AWS receives a returned device that was part of a cluster, AWS perform a complete erasure of the device. This erasure follows the National Institute of Standards and Technology (NIST) 800-88 standards.
 

Cluster

 
 

 

AWS OpsHub

The Snow Family Devices offer a user-friendly tool, AWS OpsHub for Snow Family, that users can use to manage the devices and local AWS services. Users can use AWS OpsHub on a client computer to perform tasks such as unlocking and configuring single or clustered devices, transferring files, and launching and managing instances running on Snow Family Devices. Users can use AWS OpsHub to manage both the Storage Optimized and Compute Optimized device types and the Snow device. The AWS OpsHub application is available at no additional cost.

  • AWS OpsHub takes all the existing operations available in the Snowball API and presents them as a simple graphical user interface. This interface helps you quickly and easily migrate data to the AWS Cloud and deploy edge computing applications on Snow Family Devices.
  • AWS OpsHub provides a unified view of AWS services that are running on Snow Family Devices and automates operational tasks through AWS Systems Manager.
  • With AWS OpsHub, users with different levels of technical expertise can easily manage a large number of Snow Family Devices. With just a few clicks, you can unlock devices, transfer files, manage Amazon EC2 instances, and monitor device metrics.
  • When users Snow device arrives at the site, they download, install, and launch the AWS OpsHub application on a client machine, such as a laptop. After installation, users can unlock the device and start managing it and using supported AWS services locally.
  • AWS OpsHub provides a dashboard that summarizes key metrics such as storage capacity and active instances on users device. It also provides a selection of the AWS services that are supported on the Snow Family Devices. Within minutes, users can begin transferring files to the device.
  • AWS OpsHub will automatically check for AWS OpsHub software updates when the client machine that AWS OpsHub is running on is connected to the internet. When there is a software update, users will be notified on the application and will be given the option to download and update the latest software. 

AWS OpsHub is an application that users can download from the Snowball resources page. It offers a graphical user interface for managing the AWS Snow Family devices. AWS OpsHub makes it easy to setup and manage AWS Snowball devices enabling users to rapidly deploy edge computing workloads and simplify data migration to the cloud. With just a few clicks in AWS OpsHub, users can unlock and configure devices, drag-and-drop data to devices, launch and manage EC2 instances on devices, or monitor device metrics. AWS OpsHub is available globally at no extra charge.

AWS OpsHub is an application that users can download and install on any Windows or Mac client machine, such as a laptop. Once installed AWS OpsHub and have AWS Snow Family device on site, open AWS OpsHub and unlock the device. Users will then be presented with a dashboard showing the device and its system metrics. Users can then begin deploying the edge applications or migrating data to the device with just a few clicks.

Users can use AWS OpsHub to manage and operate AWS Snow Family devices and the AWS services that run on them. AWS OpsHub is an application that runs on a local client machine, such as a laptop, and can operate in disconnected or connected environments. 

With AWS OpsHub, users can use and manage AWS services on the Snow Family Devices. Currently, AWS OpsHub supports the following resources:

  • Amazon Elastic Compute Cloud (Amazon EC2) instances—Use Amazon EC2 instances to run software installed on a virtual server without sending it to the AWS Cloud for processing.
  • AWS DataSync—Transfer a large number of files between your on-premises storage and other AWS Cloud locations, such as file systems or Amazon S3.
  • Network File System (NFS)—Use file shares to move data to your device. You can ship the device to AWS to transfer your data to the AWS Cloud, or use DataSync to other AWS Cloud locations.
 

AWS Snowball Pricing

 

 

With AWS Snowball, users pay only for the use of the device and for data transfer out of AWS. Pricing includes a service fee per job, which includes 10 days of usage onsite, and a per day fee for every additional day used the device before sending it back to AWS. The service fee and per day fee vary by AWS Region and depend on which of the four Snowball device types that users choose: Storage Optimized for Data Transfer only, Storage Optimized with EC2 Compute, Compute Optimized, or Compute Optimized with GPU.

AWS Snowball also offers significantly discounted pricing (up to 62%) for 1-year and 3-year usage commitments. For 1-year or 3-year commitments, please contact users can sales team; users cannot make this selection in the AWS Console.

Service Fee per Job

This is a one-time setup fee per job ordered through the console, and it includes ten days of onsite usage. Shipping days are not counted toward the ten days, including the day the device is received and the day it is shipped back to AWS. Pricing is listed by Region in the tables below.

Per Day Fee

The first ten days of onsite usage is included in the service fee. If the device is kept for more than ten days, users incur a fee for each day for keeping the device. The per day fee is listed by Region in the tables below.

Data Transfer

Data transfer IN to Amazon S3 is $0.00 per GB (free). Data transfer OUT of Amazon S3 is priced by Region. See data transfer OUT pricing by Region in the table below.

Shipping

Shipping charges are calculated based on standard carrier rates for the shipping location and shipping option isers choose (e.g. 2-day, overnight). The default shipping option is 2-day.

AWS Service Charges

Standard AWS service charges apply for usage incurred from importing or exporting data to or from your Amazon S3 buckets, such as Amazon S3 PUT requests or AWS KMS request charges (if KMS encryption is enabled on the Amazon S3 bucket).

Best Practices

AWS Snowball

To help get the maximum benefit and satisfaction with AWS Snowball Edge device, AWS recommend that users follow these best practices.

The following are recommendations and best practices for maintaining security while working with an AWS Snowball Edge device.

General Security
  • If noticed anything that looks suspicious about the AWS Snowball Edge device, don’t connect it to the internal network. Instead, contact AWS Support, and a new AWS Snowball Edge device will be shipped.
  • AWS recommend that users don’t save a copy of the unlock code in the same location on the workstation as the manifest for that job. Saving these in different locations helps prevent unauthorized parties from gaining access to the AWS Snowball Edge device. For example, users can save a copy of the manifest to the local server, and email the code to a user that unlocks the device. This approach limits access to the AWS Snowball Edge device to individuals who have access to files saved on the server and the user’s email address.
  • The credentials displayed when users run the Snowball client command snowballEdge credentials are a pair of keys: an access key and a secret key. These keys are only associated with the job and the local resources on the device. They don’t map to your AWS account or any other AWS account. When  trying to use these keys to access services and resources in the AWS Cloud, they will fail because they only work for the local resources associated with the job.
Network Security
  • AWS recommend that users only use one method at a time for reading and writing data to a local bucket on an AWS Snowball Edge device. Using both the file interface and the Amazon S3 Adapter for Snowball on the same Amazon S3 bucket at the same time can result in read/write conflicts. 
  • To prevent corrupting the data, don’t disconnect the AWS Snowball Edge device or change its network settings while transferring data.
  • Files that are being written to on the device should be in a static state. Files that are modified while they are being written to can result in read/write conflicts.
Resource Management

Consider the following best practices for managing jobs and resources on the AWS Snowball Edge device.

  • The 10 free days for performing on-premises data transfer start the day after the AWS Snowball Edge device arrives at users data center. This applies to all Snow Family device types.
  • The Job created status is the only status in which users can cancel a job. When a job changes to a different status, users can’t cancel the job. This applies to clusters.
  • For import jobs, don’t delete the local copies of the transferred data until the import into Amazon S3 is successful. As part of the process, be sure to verify the results of the data transfer.
Performance

Following, users can find recommendations and information about AWS Snowball Edge device performance. This section describes performance in general terms, because on-premises environments have a different way of doing things—different network technologies, different hardware, different operating systems, different procedures, and so on. Transferring smaller files reduces the transfer speed due to decreased overhead. If users have many small files, AWS recommend that users zip them up into larger archives before transferring them onto a Snowball Edge device.

Performance Recommendations

The following practices are highly recommended, because they have the largest impact on improving the performance of data transfer:

  • AWS recommend that users have no more than 500,000 files or directories within each directory.
  • AWS recommend that all files transferred to a Snowball Edge device be no smaller than 1 MB in size.
  • If users have many files smaller than 1 MB in size, AWS recommend that zip them up into larger archives before transferring them onto a Snowball Edge device.
 
Speeding Up Data Transfer

One of the major ways that users can improve the performance of an AWS Snowball Edge device is to speed up the transfer of data going to and from a device. In general, users can improve the transfer speed from data source to the device in the following ways. This following list is ordered from largest to smallest positive impact on performance:

  • Perform multiple write operations at one time:  To do this, run each command from multiple terminal windows on a computer with a network connection to a single AWS Snowball Edge device.
  • Transfer small files in batches: Each copy operation has some overhead because of encryption. To speed up the process, batch files together in a single archive. When batching files together, they can be auto-extracted when they are imported into Amazon S3. 
  • Write from multiple computers: A single AWS Snowball Edge device can be connected to many computers on a network. Each computer can connect to any of the three network interfaces at once.
  • Don’t perform other operations on files during transfer: Renaming files during transfer, changing their metadata, or writing data to the files during a copy operation has a negative impact on transfer performance. AWS recommend that files remain in a static state while transferring them.
  • Reduce local network use: AWS Snowball Edge device communicates across users local network. So you can improve data transfer speeds by reducing other local network traffic between the AWS Snowball Edge device, the switch it’s connected to, and the computer that hosts data source.
  • Eliminate unnecessary hops: AWS recommend that users set up the AWS Snowball Edge device, users data source, and the computer running the terminal connection between them so that they’re the only machines communicating across a single switch. Doing so can improve data transfer speeds.
 
 

AWS Snowball accelerates moving large amounts of data into and out of AWS using secure Snowball appliances. The Snowball appliance is purpose-built for efficient data storage and transfer. All AWS Regions have 80 TB Snowballs while US Regions have both 50 TB and 80 TB models. The Snowball appliance is rugged enough to withstand an 8.5-G jolt. At less than 50 pounds, the appliance is light enough for one person to carry. It is entirely self-contained, with a power cord, one RJ45 1 GigE and two SFP+ 10 GigE network connections on the back and an E Ink display and control panel on the front. Each Snowball appliance is water-resistant and dustproof and serves as its own rugged shipping container. AWS transfers users data directly onto and off of Snowball storage devices using Amazon’s high-speed internal network and bypasses the Internet.

  • For datasets of significant size, Snowball is often faster than Internet transfer and more cost effective than upgrading connectivity.
  • AWS Snowball supports importing data into and exporting data from Amazon S3 buckets. From there, the data can be copied or moved to other AWS services such as Amazon EBS and Amazon Glacier as desired.
  • Users can perform local data transfers between on-premises data center and a Snowball. By doing these, users can transfers through the Snowball client, a standalone downloadable client. 
  • The Snowball is its own shipping container, and its E Ink display changes to show the shipping label when the Snowball is ready to ship.
  • Users can manage jobs through the AWS Snow Family Management Console or programmatically with the job management API.