s5cmd Authentication Using Enviroment Variables

At work, I get to work with some fantastic tech that pushes the boundaries of performance. I needed to do some performance testing from a Windows server into a FlashBlade using s3. I reached out to a colleague of mine, Joshua Robinson, who told me about s5cmd. s5cmd is a very fast, parallel s3 compatible command-line client.

Check out Joshua’s post for some performance numbers. Here’s a direct quote from his post.

For uploads, s5cmd is 32x faster than s3cmd and 12x faster than aws-cli. For downloads, s5cmd can saturate a 40Gbps link (~4.3 GB/s), whereas s3cmd and aws-cli can only reach 85 MB/s and 375 MB/s respectively.

Using enviroment variables for s5cmd authentication

But I’m not here to talk about performance; I’m here to talk about authentication. s5cmd can use AWS CLI s3 like environment variables for authentication, and I want to show some config examples.

First up, the syntax to set your environment variables depends on the shell in use. Examples using both using cmd or powershell are below.

cmd

set AWS_ACCESS_KEY_ID=ABCDEFGHIJKLMNOPQRST
set AWS_SECRET_ACCESS_KEY=aBcdeFGhiJKLM/ABCDEFG/HIJKLMNOPQRSTUVWXY

PowerShell

$env:AWS_ACCESS_KEY_ID='ABCDEFGHIJKLMNOPQRST'
$env:AWS_SECRET_ACCESS_KEY='aBcdeFGhiJKLM/ABCDEFG/HIJKLMNOPQRSTUVWXY'

Once you set the environment variables, you can use s5cmd. Since I’m targeting a FlashBlade in my lab, I need to specify the --endpoint-url parameter to tell the s5cmd where my s3 bucket is. Next, I define the command I want to use, a listing with ls, then specify the bucket, which here is s3://aen-s3-bucket/. After that, you will get the output written to standard out.

s5cmd.exe --endpoint-url https://10.21.200.27/ ls s3://aen-s3-bucket/

2022/02/04 19:44:32 9175040 file1.txt
2022/02/04 19:44:32 9175040 file2.txt
2022/02/04 19:44:32 9175040 file3.txt
2022/02/04 19:44:32 9175040 file4.txt
2022/02/04 19:44:32 9175040 file5.txt