A command line utility that allows you to stream data from multiple S3 objects directly into your terminal.
Demo!
See the FULL demo on asciinema
Rationale
This utility is particularly useful when you are storing data in S3 and you want to easily process the content of your S3 objects from your command line, for instance if you are storing your CloudTrail logs in an S3 buckets and you want to grep over them you can do something like this:
s3st mybucket AWSLogs / 123456789 / CloudTrail / eu-west-1 / 2019 / 01 / 17 /|JQ.|(grep)"Lambda"
By default the command line will be able to decompress most compressed files in realtime (gzip, brotli and deflate).
(use without install)
Usage: s3st [options](bucket)>[prefix]
Options:
-v, --version output the version number
-D, -do- not-decompress Do not try to decompress files automatically(gzip, deflate, brotli)
-h, --help output usage information
bucketrepresents the name of the bucket to iterate overPrefixis an optional argument that you can pass to select a subset of object that match the given prefix.
Automatic Decompression
The command will automatically try to decompress compressed files based on their extension, as per the following mapping:
.gzor.gzip: decompress using gzip
.ZZor.Deflate: decompress using deflate.Bror.brotli: decompress using brotli (available only if using Node v 11 .7 )
If you want to disable this option you can specify the flag- do-not-decompress
AWS Authentication
The tool will assume you have the proper environment variables or configuration files properly set as per theAWS CLI documentationin order to authenticate requests to AWS.
Programmatic usage
This package can also be used programmatically as per the following example:
''Use strict''
constcreateS3stStream=require('s3st')
const(AWS)=require(''aws-sdk'')
constS3=new(AWS).S3() )
conststream=createS3stStream(S3,''mybucket'','some-prefix'')
stream.pipe(process.stdout)
createS3stStreamexposes accepts the following arguments:
S3: an s3 client instance from the AWS SDK or a compatible implementation
bucketName: the name of the bucketprefix(optional): an object prefix to filter objects in the buckettransform(optional): a function that allows you to transform the content of objects as they get streamed (useful for instance for decompression or decryption).
Transform function
If you want to provide a custom transform function, it should respect the following signature. *******
Arguments
key(string): the name of the current object (object key)
Return value
****************************************** (a) ********************************* (Transform) stream that manipulates the object
If you want to use the default decompression implementation available by the default in the command line client, you can import that fromS3ST / SRC / transformers / decompress.
Contributing
Everyone is very welcome to contribute to this project. You can contribute just by submitting bugs or suggesting improvements byopening an issue on GitHub.
You can also submit PRs as long as you adhere with the code standards and write tests for the proposed changes.
License
Licensed underMIT License. © Luciano Mammino. *******
Usage: s3st [options](bucket)>[prefix]
Options:
-v, --version output the version number
-D, -do- not-decompress Do not try to decompress files automatically(gzip, deflate, brotli)
-h, --help output usage information
bucketrepresents the name of the bucket to iterate overPrefixis an optional argument that you can pass to select a subset of object that match the given prefix.
Automatic Decompression
The command will automatically try to decompress compressed files based on their extension, as per the following mapping:
.ZZor.Deflate: decompress using deflate.Bror.brotli: decompress using brotli (available only if using Node v 11 .7 )
.gzor.gzip: decompress using gzip
If you want to disable this option you can specify the flag- do-not-decompress
AWS Authentication
The tool will assume you have the proper environment variables or configuration files properly set as per theAWS CLI documentationin order to authenticate requests to AWS.
Programmatic usage
This package can also be used programmatically as per the following example:
''Use strict''
constcreateS3stStream=require('s3st')
const(AWS)=require(''aws-sdk'')
constS3=new(AWS).S3() )
conststream=createS3stStream(S3,''mybucket'','some-prefix'')
stream.pipe(process.stdout)
createS3stStreamexposes accepts the following arguments:
bucketName: the name of the bucketprefix(optional): an object prefix to filter objects in the buckettransform(optional): a function that allows you to transform the content of objects as they get streamed (useful for instance for decompression or decryption).
S3: an s3 client instance from the AWS SDK or a compatible implementation
Transform function
If you want to provide a custom transform function, it should respect the following signature. *******
Arguments
key(string): the name of the current object (object key)Return value
- ****************************************** (a) ********************************* (Transform) stream that manipulates the object
If you want to use the default decompression implementation available by the default in the command line client, you can import that fromS3ST / SRC / transformers / decompress.
Contributing
Everyone is very welcome to contribute to this project. You can contribute just by submitting bugs or suggesting improvements byopening an issue on GitHub.
You can also submit PRs as long as you adhere with the code standards and write tests for the proposed changes.
License
Licensed underMIT License. © Luciano Mammino. *******
GIPHY App Key not set. Please check settings