To make it easy to upload and download objects from Amazon S3, we provide a TransferUtility component with built-in support for background transfers, progress tracking, and MultiPart uploads. The Transfer Utility component set includes a Service called the TransferService, which monitors network connectivity changes. When the device goes offline, the TransferService will pause all ongoing transfers; when the device is back online, the Transfer Service will resume paused transfers.
Starting with version 2.7.0, the `TransferService` will not be automatically started or stopped by `TransferUtility`. You have to start `TransferService` manually from your application. A recommended way is to start the service upon Application startup, by including the following line in the `onCreate` method of your app's Application class.
```java
getApplicationContext().startService(new Intent(getApplicationContext(), TransferService.class));
```
This section explains how to implement upload and download functionality and a number of additional storage use cases.
<Callout>
**Note:** If you use the transfer utility MultiPart upload feature, take advantage of automatic cleanup features by setting up the [AbortIncompleteMultipartUpload](https://docs.aws.amazon.com/AmazonS3/latest/dev/intro-lifecycle-rules.html) action in your Amazon S3 bucket life cycle configuration.
</Callout>
## Upload a File
The following example shows how to use the TransferUtility to upload a file. Instantiate the TransferUtility object using the provided TransferUtility builder function. Use the `AWSMobileClient` to get the `AWSConfiguration` and `AWSCredentialsProvider` to pass into the builder. See [Authentication](/sdk/auth/getting-started) for more details.
The TransferUtility checks the size of the file being uploaded and automatically switches over to using multi-part uploads if the file size exceeds 5 MB.
```java
import android.app.Activity;
import android.content.Intent;
import android.os.Bundle;
import android.util.Log;
import com.amazonaws.mobile.client.AWSMobileClient;
import com.amazonaws.mobile.client.Callback;
import com.amazonaws.mobile.client.UserStateDetails;
import com.amazonaws.mobileconnectors.s3.transferutility.TransferListener;
import com.amazonaws.mobileconnectors.s3.transferutility.TransferObserver;
import com.amazonaws.mobileconnectors.s3.transferutility.TransferService;
import com.amazonaws.mobileconnectors.s3.transferutility.TransferState;
import com.amazonaws.mobileconnectors.s3.transferutility.TransferUtility;
import com.amazonaws.services.s3.AmazonS3Client;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
public class MainActivity extends Activity {
private static final String TAG = MainActivity.class.getSimpleName();
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
getApplicationContext().startService(new Intent(getApplicationContext(), TransferService.class));
// Initialize the AWSMobileClient if not initialized
AWSMobileClient.getInstance().initialize(getApplicationContext(), new Callback<UserStateDetails>() {
@Override
public void onResult(UserStateDetails userStateDetails) {
Log.i(TAG, "AWSMobileClient initialized. User State is " + userStateDetails.getUserState());
uploadWithTransferUtility();
}
@Override
public void onError(Exception e) {
Log.e(TAG, "Initialization error.", e);
}
});
}
public void uploadWithTransferUtility() {
TransferUtility transferUtility =
TransferUtility.builder()
.context(getApplicationContext())
.awsConfiguration(AWSMobileClient.getInstance().getConfiguration())
.s3Client(new AmazonS3Client(AWSMobileClient.getInstance()))
.build();
File file = new File(getApplicationContext().getFilesDir(), "sample.txt");
try {
BufferedWriter writer = new BufferedWriter(new FileWriter(file));
writer.append("Howdy World!");
writer.close();
}
catch(Exception e) {
Log.e(TAG, e.getMessage());
}
TransferObserver uploadObserver =
transferUtility.upload(
"public/sample.txt",
new File(getApplicationContext().getFilesDir(),"sample.txt"));
// Attach a listener to the observer to get state update and progress notifications
uploadObserver.setTransferListener(new TransferListener() {
@Override
public void onStateChanged(int id, TransferState state) {
if (TransferState.COMPLETED == state) {
// Handle a completed upload.
}
}
@Override
public void onProgressChanged(int id, long bytesCurrent, long bytesTotal) {
float percentDonef = ((float) bytesCurrent / (float) bytesTotal) * 100;
int percentDone = (int)percentDonef;
Log.d(TAG, "ID:" + id + " bytesCurrent: " + bytesCurrent
+ " bytesTotal: " + bytesTotal + " " + percentDone + "%");
}
@Override
public void onError(int id, Exception ex) {
// Handle errors
}
});
// If you prefer to poll for the data, instead of attaching a
// listener, check for the state and progress in the observer.
if (TransferState.COMPLETED == uploadObserver.getState()) {
// Handle a completed upload.
}
Log.d(TAG, "Bytes Transferred: " + uploadObserver.getBytesTransferred());
Log.d(TAG, "Bytes Total: " + uploadObserver.getBytesTotal());
}
}
```
If you run this code, login to your AWS console, and go to the S3 service, you'll see a bucket and file structure like this (in this example the friendly name specified was `dev` and the bucket name was `storagedemo`):
## Download a File
The following example shows how to use the TransferUtility to download a file. Instantiate the TransferUtility object using the provided TransferUtility builder function. Use the `AWSMobileClient` to get the `AWSConfiguration` and `AWSCredentialsProvider` to pass into the builder. See [Authentication](/sdk/auth/getting-started) for more details.
```java
import android.app.Activity;
import android.content.Intent;
import android.os.Bundle;
import android.util.Log;
import com.amazonaws.mobile.client.AWSMobileClient;
import com.amazonaws.mobile.client.Callback;
import com.amazonaws.mobile.client.UserStateDetails;
import com.amazonaws.mobileconnectors.s3.transferutility.TransferListener;
import com.amazonaws.mobileconnectors.s3.transferutility.TransferObserver;
import com.amazonaws.mobileconnectors.s3.transferutility.TransferService;
import com.amazonaws.mobileconnectors.s3.transferutility.TransferState;
import com.amazonaws.mobileconnectors.s3.transferutility.TransferUtility;
import com.amazonaws.services.s3.AmazonS3Client;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
public class MainActivity extends Activity {
private static final String TAG = MainActivity.class.getSimpleName();
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
getApplicationContext().startService(new Intent(getApplicationContext(), TransferService.class));
// Initialize the AWSMobileClient if not initialized
AWSMobileClient.getInstance().initialize(getApplicationContext(), new Callback<UserStateDetails>() {
@Override
public void onResult(UserStateDetails userStateDetails) {
Log.i(TAG, "AWSMobileClient initialized. User State is " + userStateDetails.getUserState());
downloadWithTransferUtility();
}
@Override
public void onError(Exception e) {
Log.e(TAG, "Initialization error.", e);
}
});
}
private void downloadWithTransferUtility() {
TransferUtility transferUtility =
TransferUtility.builder()
.context(getApplicationContext())
.awsConfiguration(AWSMobileClient.getInstance().getConfiguration())
.s3Client(new AmazonS3Client(AWSMobileClient.getInstance()))
.build();
TransferObserver downloadObserver =
transferUtility.download(
"public/sample.txt",
new File(getApplicationContext().getFilesDir(), "download.txt"));
// Attach a listener to the observer to get state update and progress notifications
downloadObserver.setTransferListener(new TransferListener() {
@Override
public void onStateChanged(int id, TransferState state) {
if (TransferState.COMPLETED == state) {
// Handle a completed upload.
}
}
@Override
public void onProgressChanged(int id, long bytesCurrent, long bytesTotal) {
float percentDonef = ((float)bytesCurrent/(float)bytesTotal) * 100;
int percentDone = (int)percentDonef;
Log.d("Your Activity", " ID:" + id + " bytesCurrent: " + bytesCurrent + " bytesTotal: " + bytesTotal + " " + percentDone + "%");
}
@Override
public void onError(int id, Exception ex) {
// Handle errors
}
});
// If you prefer to poll for the data, instead of attaching a
// listener, check for the state and progress in the observer.
if (TransferState.COMPLETED == downloadObserver.getState()) {
// Handle a completed upload.
}
Log.d("Your Activity", "Bytes Transferred: " + downloadObserver.getBytesTransferred());
Log.d("Your Activity", "Bytes Total: " + downloadObserver.getBytesTotal());
}
}
```
## Track Transfer Progress
With the TransferUtility, the `download` and `upload` methods return a `TransferObserver` object. This object gives access to:
1. The transfer state, as an `enum`
2. The total bytes that have been transferred so far
3. The total bytes remaining to transfer
4. A unique ID that you can use to keep track of each transfer
Given the transfer ID, the `TransferObserver` object can be retrieved from anywhere in your app, even if the app was terminated during a transfer. It also lets you create a `TransferListener`, which will be updated on changes to transfer state, progress, and when an error occurs.
To get the progress of a transfer, call `setTransferListener()` on your `TransferObserver`. This requires you to implement `onStateChanged`, `onProgressChanged`, and `onError` as shown in the example.
```java
TransferObserver transferObserver = download(MY_BUCKET, OBJECT_KEY, MY_FILE);
transferObserver.setTransferListener(new TransferListener(){
@Override
public void onStateChanged(int id, TransferState state) {
// do something
}
@Override
public void onProgressChanged(int id, long bytesCurrent, long bytesTotal) {
int percentage = (int) (bytesCurrent/bytesTotal * 100);
//Display percentage transferred to user
}
@Override
public void onError(int id, Exception ex) {
// do something
}
});
```
The transfer ID can be retrieved from the `TransferObserver` object that is returned from the upload or download function. You can also query for `TransferObservers` using the `getTransfersWithType(transferType)` or the `getTransfersWithTypeAndState(transferType, transferState)` method.
```java
// Gets id of the transfer.
int transferId = transferObserver.getId();
```
## Pause a Transfer
Transfers can be paused using the `pause(transferId)` method. If your app is terminated, crashes, or loses Internet connectivity, transfers are automatically paused.
To pause a single transfer:
```java
transferUtility.pause(idOfTransferToBePaused);
```
To pause all uploads:
```java
transferUtility.pauseAllWithType(TransferType.UPLOAD);
```
To pause all downloads:
```java
transferUtility.pauseAllWithType(TransferType.DOWNLOAD);
```
To pause all transfers of any type:
```java
transferUtility.pauseAllWithType(TransferType.ANY);
```
## Resume a Transfer
In the case of a loss in network connectivity, transfers will automatically resume when network connectivity is restored. If the app crashed or was terminated by the operating system, transfers can be resumed with the `resume(transferId)` method.
To resume a single transfer:
```java
transferUtility.resume(idOfTransferToBeResumed);
```
To resume all uploads:
```java
transferUtility.resumeAllWithType(TransferType.UPLOAD);
```
To resume all downloads:
```java
transferUtility.resumeAllWithType(TransferType.DOWNLOAD);
```
To resume all transfers of any type:
```java
transferUtility.resumeAllWithType(TransferType.ANY);
```
## Cancel a Transfer
To cancel an upload, call cancel() or cancelAllWithType() on the `TransferUtility` object.
To cancel a single transfer, use:
```java
transferUtility.cancel(idToBeCancelled);
```
To cancel all transfers of a certain type, use:
```java
transferUtility.cancelAllWithType(TransferType.DOWNLOAD);
```
## Background Transfers
The SDK uploads and downloads objects from Amazon S3 using background threads. These transfers will continue to run regardless of whether your app is running in the foreground or background.
## Long-running Transfers
When you want your app to perform long-running transfers in the background, you can initiate the transfers from a background service that you can implement within your app. A recommended way to use a service to initiate the transfer is demonstrated in the [Transfer Utility sample application](https://github.com/awslabs/aws-sdk-android-samples/tree/master/S3TransferUtilitySample).
## Supporting TransferService on Oreo and above
`TransferNetworkLossHandler`, a broadcast receiver that listens for network connectivity changes is introduced in `2.11.0`. `TransferNetworkLossHandler` pauses the on-going transfers when the network goes offline and resumes the transfers that were paused when the network comes back online. `TransferService` registers the `TransferNetworkLossHandler` when the service is created and de-registers the handler when the service is destroyed.
* `TransferService` will be moved to the foreground state when the device is running Android Oreo (API Level 26) and above.
* Transitioning to the foreground state requires a valid on-going `Notification` object, identifier for on-going notification and the flag that determines the ability to remove the on-going notification when the service transitions out of foreground state. If a valid notification object is not passed in, the service will not be transitioned into the foreground state.
* The `TransferService` can now be started using `startForegroundService` method to move the service to foreground state. The service can be invoked in the following way to transition the service to foreground state.
```java
Intent tsIntent = new Intent(getApplicationContext(), TransferService.class);
tsIntent.putExtra(TransferService.INTENT_KEY_NOTIFICATION, <notification-object>);
tsIntent.putExtra(TransferService.INTENT_KEY_NOTIFICATION_ID, <notification-id>);
tsIntent.putExtra(TransferService.INTENT_KEY_REMOVE_NOTIFICATION, <remove-notification-when-service-stops-foreground>);
getApplicationContext().startForegroundService(tsIntent);
```
## Supporting Unicode characters in key-names
**Upload/download objects**
* Since `2.4.0` version of the SDK, the key name containing characters that require special handling are URL encoded and escaped `( space, %2A, ~, /, :, ', (, ), !, [, ] )` by the `AmazonS3Client`, after which the AWS Android Core Runtime encodes the URL resulting in double encoding of the key name.
* Starting `2.11.0`, the additional layer of encoding and escaping done by `AmazonS3Client` is removed. The key name will not be encoded and escaped by `AmazonS3Client`. Now, the key name that is given to `AmazonS3Client` or `TransferUtility` will appear on the Amazon S3 console as is.
**List Objects**
* When a S3 bucket contains objects with key names containing characters that require special handling, and since the SDK has an XML parser, (XML 1.0 parser) which cannot parse some characters, the SDK is required to request that Amazon S3 encode the keys in the response. This can be done by passing in `url` as `encodingType` in the `ListObjectsRequest`.
```java
AmazonS3Client s3 = new AmazonS3Client(credentials);
final ObjectListing objectListing = s3.listObjects(
new ListObjectsRequest(bucketName, prefix, null, null, null)
.withEncodingType(Constants.URL_ENCODING));
```
* Since `2.4.0`, there was a bug where the SDK did not decode the key names which are encoded by S3 when `url` is requested as the `encodingType`. This is fixed in `2.11.0`, where the SDK will decode the key names in the `ListObjectsResponse` sent by S3.
* If you have objects in S3 bucket that has a key name containing characters that require special handling, you need to pass the `encodingType` as `url` in the `ListObjectsRequest`.
## Transfer with Object Metadata
To upload a file with metadata, use the `ObjectMetadata` object. Create a `ObjectMetadata` object and add in the metadata headers and pass it to the upload function.
```java
import com.amazonaws.services.s3.model.ObjectMetadata;
ObjectMetadata myObjectMetadata = new ObjectMetadata();
//create a map to store user metadata
Map<String, String> userMetadata = new HashMap<String,String>();
userMetadata.put("myKey","myVal");
//call setUserMetadata on your ObjectMetadata object, passing it your map
myObjectMetadata.setUserMetadata(userMetadata);
```
Then, upload an object along with its metadata:
```java
TransferObserver observer = transferUtility.upload(
MY_BUCKET, /* The bucket to upload to */
OBJECT_KEY, /* The key for the uploaded object */
MY_FILE, /* The file where the data to upload exists */
myObjectMetadata /* The ObjectMetadata associated with the object*/
);
```
To download the metadata, use the S3 `getObjectMetadata` method. See the [API Reference](http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3Client.html#getObjectMetadata%28com.amazonaws.services.s3.model.GetObjectMetadataRequest%29) and [Object Key and Metadata](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html) for more information.
## Transfer Utility Options
You can use the `TransferUtilityOptions` object to customize the operations of the TransferUtility.
### TransferThreadPoolSize
This parameter allows you to specify the number of transfers that can run in parallel. By increasing the number of threads, you will be able to increase the number of parts of a multi-part upload that will be uploaded in parallel. By default, this is set to 2 * (N + 1), where N is the number of available processors on the mobile device. The minimum allowed value is 2.
```java
TransferUtilityOptions options = new TransferUtilityOptions();
options.setTransferThreadPoolSize(8);
TransferUtility transferUtility = TransferUtility.builder()
// Pass-in S3Client, Context, AWSConfiguration/DefaultBucket Name
.transferUtilityOptions(options)
.build();
```
### TransferNetworkConnectionType
The `TransferNetworkConnectionType` option allows you to restrict the type of network connection (WiFi / Mobile / ANY) over which the data can be transferred to Amazon S3.
```java
TransferUtilityOptions options = new TransferUtilityOptions(10, TransferNetworkConnectionType.WIFI);
TransferUtility transferUtility = TransferUtility.builder()
// Pass-in S3Client, Context, AWSConfiguration/DefaultBucket Name
.transferUtilityOptions(options)
.build();
```
By specifying `TransferNetworkConnectionType.WIFI` , data transfers to and from S3 will only happen when the device is on a WiFi connection