I have been performing some data transformations using Lambda and S3 Events and for certain S3 keys I noticed that I was getting an Error 403 in my code:
[ERROR] ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden
Traceback (most recent call last):
File "/var/task/index.py", line 159, in handler
if s3_obj.content_length < (400 * MB):
File "/var/runtime/boto3/resources/factory.py", line 339, in property_loader
self.load()
File "/var/runtime/boto3/resources/factory.py", line 505, in do_action
response = action(self, *args, **kwargs)
File "/var/runtime/boto3/resources/action.py", line 83, in __call__
response = getattr(parent.meta.client, operation_name)(**params)
File "/var/runtime/botocore/client.py", line 320, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/var/runtime/botocore/client.py", line 623, in _make_api_call
raise error_class(parsed_response, operation_name)
The interesting thing that came from this was that the error seemed to specifically happen on files that contained special (i.e. non-alphanumeric) characters. Research indicated that the characters were actually valid - see https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html - so it isn’t an issue with the key name itself.
It turns out that the issue is with the fact that when the S3 Event returns the queue it is URL encoded but when you attempt to use the key in boto3 then it fails. The solution is to do the following:
from urllib import parse
import boto3
S3_RESOURCE = boto3.resource('s3')
def handler(event, context): # pylint: disable=unused-argument
"""
Lambda entry point.
"""
for record in event["Records"]:
# Lambda struggles to process a file over 400Mb, so am not going to process files larger
# than that.
bucket = record["s3"]["bucket"]["name"]
key = parse.unquote(record["s3"]["object"]["key"])
s3_obj = S3_RESOURCE.Object(bucket, key)
The s3_obj above will then start working.
Website : https://michael.ransley.co