Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[S3] Feature Request: Support S3 Access Points #1450

Open
melya opened this issue Jun 21, 2023 · 11 comments
Open

[S3] Feature Request: Support S3 Access Points #1450

melya opened this issue Jun 21, 2023 · 11 comments

Comments

@melya
Copy link
Contributor

melya commented Jun 21, 2023

Hey 👋

Official aws-php-sdk supports presign of s3 access points, here is an example

$command = $awsSdk->getCommand(
   "GetObject",
   [
      "Bucket" => "arn:aws:s3-object-lambda:eu-central-1:000000000000:accesspoint/<accesspoint-name>",
      "Key" => "<object-key>"
   ]
);

$preSignedUrl = $awsSdk->createPresignedRequest($command, "+5 min")->getUri();

So, the bucket like arn:aws:s3-object-lambda:eu-central-1:000000000000:accesspoint/<accesspoint-name>
resolves into https://<accesspoint-name>-000000000000.s3-object-lambda.eu-central-1.amazonaws.com


*Real use-case: s3 lambda func generates a thumbnail by accessing the pre-signed URL

Would be great to have this functionality inside async-aws ❤️

Research:

@jderusse
Copy link
Member

did you tried https://async-aws.com/features/presign.html ?

@melya
Copy link
Contributor Author

melya commented Jun 22, 2023

Sure, here is an example with using async-aws/s3

$client->presign(\AsyncAws\S3\Input\GetObjectRequest::create([
   "Bucket" => "arn:aws:s3-object-lambda:eu-central-1:000000000000:accesspoint/thumbnail-generator",
   "Key" => "Foo/HappyFace.jpg"
]));

Result of async-aws

https://s3.eu-central-1.amazonaws.com/arn%3Aaws%3As3-object-lambda%3Aeu-central-1%3A000000000000%3Aaccesspoint%2Fthumbnail-generator/Foo/HappyFace.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230622T102450Z&X-Amz-Expires=60&X-Amz-Credential=<accessKeySignature>&x-amz-content-sha256=UNSIGNED-PAYLOAD&X-Amz-SignedHeaders=host&X-Amz-Signature=c0fdd97e5b43239ef754d49304ffc9835617292e3c772bc67e0ba45ae796e099

Result of aws-sdk-php (expected)

https://thumbnail-generator-000000000000.s3-object-lambda.eu-central-1.amazonaws.com/Foo/HappyFace.jpg?X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<accessKeySignature>&X-Amz-Date=20230622T102816Z&X-Amz-SignedHeaders=host&X-Amz-Expires=60&X-Amz-Signature=d25fc4c6c27ac025152b7691023607e7923bf22d5f503910497c74e53df5eb46

*Here is a original aws BucketEndpointArnMiddleware where actually hostname resolves depending on a Bucket

@melya
Copy link
Contributor Author

melya commented Jun 23, 2023

This is last feature I miss to fully switch to async-aws, so I open for contribution )
I need some help here to understand how would you like to implement this from helicopter (architecture) view

@melya
Copy link
Contributor Author

melya commented Jun 23, 2023

Kinda MVP with lots of hacks, but generates valid pre-signed url

<?php

declare(strict_types=1);

use AsyncAws\Core\Request;
use AsyncAws\Core\RequestContext;
use AsyncAws\Core\Stream\StreamFactory;
use AsyncAws\S3\Signer\SignerV4ForS3;

final class MVP
{
    private const ARN_PREFIX = "arn:aws:s3-object-lambda:";

    public function __construct(private readonly \AsyncAws\S3\S3Client $client)
    {
    }

    /**
     * @param array{
     *     Bucket: string,
     *     Key: string,
     *     ResponseContentDisposition?: string,
     *     ResponseContentType?: string,
     * } $input
     */
    public function __invoke(array $input, \DateTimeImmutable $expires): string
    {
        $this->assertSupports($input["Bucket"]);

        [$host, $region] = $this->arnToHost($input["Bucket"]);

        $headers = ['content-type' => 'application/xml'];
        if (isset($input["ResponseContentDisposition"])) {
            $query['response-content-disposition'] = $input["ResponseContentDisposition"];
        }

        if (isset($input["ResponseContentType"])) {
            $query['response-content-type'] = $input["ResponseContentType"];
        }

        $endpoint = "{$host}/"
            . \str_replace('%2F', '/', \rawurlencode(\ltrim($input["Key"], "/")))
            . "?"
            . \http_build_query($query, '', '&', \PHP_QUERY_RFC3986);

        $request = new Request('GET', '', $query, $headers, StreamFactory::create(''));
        $request->setEndpoint($endpoint);

        $r = new \ReflectionProperty(\AsyncAws\Core\AbstractApi::class, "credentialProvider");
        $r->setAccessible(true);
        $credentialProvider = $r->getValue($this->client);

        if (null !== $credentials = $credentialProvider?->getCredentials($this->client->getConfiguration())) {
            $signer = new SignerV4ForS3("s3-object-lambda", $region);
            $signer->presign($request, $credentials, new RequestContext(['expirationDate' => $expires]));
        }

        return $request->getEndpoint();
    }

    private function assertSupports(string $arn): void
    {
        if (0 !== \strpos($arn, self::ARN_PREFIX)) {
            throw new \LogicException("Only s3-object-lambda can be provided");
        }
    }

    /** @return array{string, string} */
    private function arnToHost(string $arn): array
    {
        $arn   = \substr($arn, \strlen(self::ARN_PREFIX));
        $parts = \explode(":", $arn, 3);
        if (3 !== \count($parts)) {
            throw new \LogicException("Invalid s3-object-lambda ARN provided");
        }
        $region      = $parts[0];
        $accountId   = $parts[1];
        $accessPoint = \explode("/", $parts[2], 2)[1] ?? $parts[2];

        return ["https://{$accessPoint}-{$accountId}.s3-object-lambda.{$region}.amazonaws.com", $region];
    }
}

@melya melya changed the title [S3] Feature Request: Use presigned URL with Multi-Region Access Points [S3] Feature Request: Presign S3 Access Points Jun 24, 2023
@melya
Copy link
Contributor Author

melya commented Jun 27, 2023

Hi @jderusse!
I've collected all needed aws documentation in the description (below Research)

Looking forward to your response. Thanks!

@melya melya changed the title [S3] Feature Request: Presign S3 Access Points [S3] Feature Request: Support S3 Access Points Jun 28, 2023
@stof
Copy link
Member

stof commented Jun 29, 2023

Have you tried using the access endpoint alias instead of it's ARN ? From my quick reading of the documentation links you provided, I think it might work already.

@melya
Copy link
Contributor Author

melya commented Jun 30, 2023

Have you tried using the access endpoint alias instead of it's ARN ? From my quick reading of the documentation links you provided, I think it might work already.

Thanks for your reply!
Yeah, alias works good 👍

Looking back to my feature request - it should be converted into support of Access Points (e.g. Outposts, ObjectLambdaAccessPoint, etc.) out of the box.

Good idea here to mention in the documentation that currently there is no support of S3 Access Points to avoid further issues and misunderstanding


Also, in my scenario with thumbnail generator - I provide a custom query parameter size which is used to generate thumbnails on the fly with needed size.
Currently, there is no ability to do this with GetObjectRequest - open for ideas how this could be handled 👀

I've made a workaround and created own GetObjectRequest which accepts CustomQuery?: array<string, string>

final class GetObjectRequest extends Input
{
    /**
     * @param array{
     *   Bucket?: string,
     *   Key?: string,
     *   ...
     *   CustomQuery?: array<string, string>,
     *
     *   @region?: string,
     * } $input
     */
    public function __construct(array $input = [])
    ... 
    public function request(): Request
    {
       ...
       $query += $this->customQuery ?? [];
       
       return new Request('GET', $uriString, $query, $headers, StreamFactory::create($body));
    }
}

@stof
Copy link
Member

stof commented Jun 30, 2023

Is passing custom query parameters supported by the official SDK ?

@stof
Copy link
Member

stof commented Jun 30, 2023

Good idea here to mention in the documentation that currently there is no support of S3 Access Points to avoid further issues and misunderstanding

PRs updating the documentation are welcome.

@melya
Copy link
Contributor Author

melya commented Jun 30, 2023

Is passing custom query parameters supported by the official SDK ?

Unfortunately No :)
With official php-sdk it can be done only by adding custom middleware which will append the query later
But in async-aws there is no such possibility at all

@melya
Copy link
Contributor Author

melya commented Jun 30, 2023

Good idea here to mention in the documentation that currently there is no support of S3 Access Points to avoid further issues and misunderstanding

PRs updating the documentation are welcome.

👍 I will create a PR later

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants