[SalesForce] Cannot list objects using AWS S3

For some reason, I cannot list objects within my S3 bucket. Here's the steps that I've taken:

I've viewed this question and copied this gist from sfdcfox into my org.

Here's my AWSS3_GetService sample implementation:

// Example implementation as follows:
public class AWSS3_GetService extends AWS {
    public override void init() {
        endpoint = new Url('https://s3.amazonaws.com/');
        region = 'us-east-1';
        service = 's3';
        accessKey = 'my-access-key';
        method = HttpMethod.XGET;
        //  Remember to set "payload" here if you need to specify a body
        //  payload = Blob.valueOf('some-text-i-want-to-send');
        //  This method helps prevent leaking secret key, 
        //  as it is never serialized
        createSigningKey('my-secret-key');
    }

    public String[] listBuckets(String resource) {
        this.resource = resource;
        setQueryParam('list-type','2');
        HttpResponse response = sendRequest();

        for (String key : response.getHeaderKeys()) {
            if (key != null) 
                System.debug(logginglevel.error, key + ': ' + response.getHeader(key));
        }
        System.debug(loggingLevel.error, 'body: ' + response.getBody());

        return null;
    }
}

Unfortunately, calling the following code does not return me the contents of the bucket:

awss3_getservice service = new awss3_getservice();
service.init();
service.listBuckets('/ti-use1-da-data-dropzone/ti-int-boss-pos/');

I can access the contents via the aws command line utility, so I don't think it's a permissions issue:

$ aws s3 ls s3://ti-use1-da-data-dropzone/ti-int-boss-pos/
                           PRE uploads/
2016-10-28 13:52:26          0
2016-10-31 16:35:33       1418 Deployment_notes.txt

I can download files as long as I have an exact name, however, I cannot list the files. Is there anything I am doing wrong?

Here is the output from my anonymous apex btw:

17:20:13.4 (798743870)|USER_DEBUG|[193]|ERROR|System.HttpRequest[Endpoint=https://s3.amazonaws.com/ti-use1-da-data-dropzone/ti-int-boss-pos/?list-type=2, Method=GET]
17:20:14.13 (1013525242)|USER_DEBUG|[195]|ERROR|valid codes: {200}, real code: 200
17:20:14.13 (1013577565)|USER_DEBUG|[196]|ERROR|response: System.HttpResponse[Status=OK, StatusCode=200]
17:20:14.13 (1013741266)|USER_DEBUG|[24]|ERROR|Accept-Ranges: bytes
17:20:14.13 (1013785482)|USER_DEBUG|[24]|ERROR|Server: AmazonS3
17:20:14.13 (1013820508)|USER_DEBUG|[24]|ERROR|ETag: "e-tag"
17:20:14.13 (1013853023)|USER_DEBUG|[24]|ERROR|Last-Modified: Fri, 28 Oct 2016 17:52:26 GMT
17:20:14.13 (1013887157)|USER_DEBUG|[24]|ERROR|x-amz-request-id: req-id
17:20:14.13 (1013922045)|USER_DEBUG|[24]|ERROR|Content-Length: 0
17:20:14.13 (1013956514)|USER_DEBUG|[24]|ERROR|x-amz-server-side-encryption: AES256
17:20:14.13 (1013989661)|USER_DEBUG|[24]|ERROR|x-amz-id-2: krcnKD4oi7ebv/amz-id+Q1uu5qaVJaGjh17UjmJOIMJSkiPvNKAgU=
17:20:14.13 (1014023304)|USER_DEBUG|[24]|ERROR|x-amz-version-id: OII3Ojhyg.vers-id
17:20:14.13 (1014056408)|USER_DEBUG|[24]|ERROR|Date: Mon, 31 Oct 2016 21:20:15 GMT
17:20:14.13 (1014090063)|USER_DEBUG|[24]|ERROR|Content-Type: binary/octet-stream

Best Answer

Starting from my prior gist, I then built a core class that is designed to work with S3 calls. Here it goes:

abstract class Core extends AWS {
    Core() {
        super();
    }

    protected S3User getChildNodeUser(Dom.XmlNode node, String ns, String name) {
        S3User result = new S3User();
        Dom.XmlNode ownerNode = node.getChildElement(name, ns);
        if(ownerNode != null) {
            result.Id = getChildNodeText(node, ns, 'ID');
            result.DisplayName = getChildNodeText(node, ns, 'DisplayName');
        }
        return result;
    }

    protected virtual override void init() {
        AmazonS3__c configSettings = AmazonS3__c.getOrgDefaults();
        endpoint = new Url('https://'+configSettings.Endpoint__c);
        accessKey = configSettings.AccessKey__c;
        region = configSettings.Region__c;
        service = 's3';
        setHeader('date', requestTime.formatGmt('E, dd MMM yyyy HH:mm:ss z'));
        //  Prevent leaking the secret key by only exposing the signing key
        createSigningKey(configSettings.SecretKey__c);
    }
}

From there, I needed a place to hold the results from the call:

public class BucketGetListObjectsResult {
    public String name, prefix, marker, delimiter;
    public Integer maxKeys;
    public Boolean isTruncated;
    public File[] files = new File[0];
    public String[] commonPrefixes = new String[0];
}

And a class to represent a File:

public class File {
    File() {

    }
    File(String bucketName) {
        bucket = bucketName;
    }
    String bucket;
    public String name, eTag;
    public S3User owner;
    public DateTime lastModified;
    public Integer size;
    public Blob contents;
}

And a class that represented a S3 user:

public class S3User {
    public String ID, DisplayName;
}

Finally, I built a class that lets me specify the various actions and get the results:

public class BucketGetListObjects extends Core {
    String bucket;
    BucketGetListObjects(String bucketName) {
        super();
        bucket = bucketName;
    }
    public BucketGetListObjects delimiter(String delimiter) {
        setQueryParam('delimiter', delimiter);
        return this;
    }
    public BucketGetListObjects marker(String marker) {
        setQueryParam('marker', marker);
        return this;
    }
    public BucketGetListObjects maxKeys(Integer maxKeys) {
        setQueryParam('max-keys', String.valueOf(maxKeys));
        return this;
    }
    public BucketGetListObjects prefix(String prefix) {
        setQueryParam('prefix', prefix);
        return this;
    }
    public override void init() {
        super.init();
        host = bucket+'.'+endpoint.getHost();
        resource = '/';
    }
    public BucketGetListObjectsResult execute() {
        method = HttpMethod.XGET;
        BucketGetListObjectsResult result = new BucketGetListObjectsResult();
        HttpResponse response = sendRequest();
        Dom.XmlNode rootNode = response.getBodyDocument().getRootElement();
        String ns = rootNode.getNamespace();
        result.Name = getChildNodeText(rootNode, ns, 'Name');
        result.Prefix = getChildNodeText(rootNode, ns, 'Prefix');
        result.Marker = getChildNodeText(rootNode, ns, 'Marker');
        result.maxKeys = getChildNodeInteger(rootNode, ns, 'MaxKeys');
        result.Delimiter = getChildNodeText(rootNode, ns, 'Delimiter');
        result.isTruncated = getChildNodeBoolean(rootNode, ns, 'IsTruncated');
        Dom.XmlNode child;
        while((child = rootNode.getChildElement('Contents', ns)) != null) {
            File file = new File();
            file.bucket = bucket;
            file.name = getChildNodeText(child, ns, 'Key');
            file.lastModified = getChildNodeDateTime(child, ns, 'LastModified');
            file.eTag = getChildNodeText(child, ns, 'ETag');
            file.size = getChildNodeInteger(child, ns, 'Size');
            file.owner = getChildNodeUser(child, ns, 'Owner');
            result.files.add(file);
            rootNode.removeChild(child);
        }
        while((child = rootNode.getChildElement('CommonPrefixes', ns)) != null) {
            result.commonPrefixes.add(getChildNodeText(child, ns, 'Prefix'));
            rootNode.removeChild(child);
        }
        return result;
    }
}

The actual class is used like this (these classes are all in a class called AWSS3):

AWSS3.Service service = new AWSS3.Service();
AWSS3.Bucket bucket = service.getBucketByName(bucketName);
AWSS3.BucketGetListObjects listObjects = bucket.bucketGetListObjects();
AWSS3.BucketGetListObjectsResults results = listObjects.execute();

The remaining parameters let you specify things like a "marker" (necessary for iterating over pages), a prefix (which lets you find related files), and so on. You'll find more details at Bucket GET.

You are also allowed to daisy-chain the calls in my design:

AWSS3.BucketGetListObjectsResults results = 
   new AWSS3.Service()
   .getBucketByName(bucketName)
   .bucketGetListObjects()
   .execute();

I can see that you're using v2 of this API, which does include the "list-type=2" parameter. If you decide to use v2, you'll need to modify my example code above to support the continuation-token, which is one of the primary differences between v2 and v1 (which is what my code demonstrates). Most of the remaining code should work verbatim.

However, to get to the final point, your request really should look something like "https://s3.amazonaws.com/?list-type=2&prefix=/ti-use1-da-data-dropzone/ti-int-boss-pos/", which is clearly not the same as what you attempted to do.

I realize that I've also left out some code, but this thing was already getting pretty long as it is. Bucket and Service are just more uninteresting wrappers that perform various actions. Service can list all of an account's buckets, for example, Bucket can list and delete file contents, and File can also be for file uploads. There's a lot of framework that went in to this class, so I tried to keep it relevant.

Related Topic