When processing data on the cloud for somebody else (e.g. a customer) or you want to share large amounts of data for other reasons, a good option is to directly use cloud storage for this. Besides DropBox (which requires a contract with a monthly fee for this kind of data) and other offerings, Amazon provides S3: You already get 5 GB of free S3 storage as part of the “Free Tier“, for anything else you only pay for what you use. At the current price(August 2017) you pay $0.023 per GB for standard storage.Additionally you have to pay for access operations: PUT, COPY, POST, or LIST Requests $0.005 per 1,000 requests, GET and all other Requests $0.004 per 10,000 requests. Data transfers between S3 buckets or from S3 to any Amazon cloud service(s) within the same region are free, to other regions or download via the internet incurr additional prices. Please check back on the Amazon pages for curent prices and conditions.
Using the excellent Walk-through example from AWS, here is a condensed version of the set-up that worked for me. I needed to process large data files for a customer, let’s call them DataHeros Ltd. here, downloading to my machine and uploading to the cloud would have been highly inefficient. FTP or other access to the customer data was not possible.
The main steps are:
- Create IAM profile for the customer
– Sign in to AWS, create a new IAM user (let’s call him/her “data-heros-ltd”) with password but no permissions yet.
– Also create a group (called “customers”)
- Create S3 bucket with folder for the customer
– Go to the S3 console.
– Create a bucket “my-company-bucket” and a folder “customer-A”, “customer-B”, etc. within “customer-A” I create a subfolder “DataHeros-Ltd”. All customers will see the initial listing, but they don’t need to see the actual names of other customer. At the same time the customer is sure that the second-level folder is the correct one for his/her data.
Let’s also create a folder “other-data” for just that, where only I have access to.
- Set permission to allow access to this area only
All customers (group) need list acess to the bucket, the current customer DataHeros-Ltd needs write access to the “customer-A / DataHeros-Ltd” area.
The permissions are set using:
– the inline policy of the user data-heros-ltd: Policy 1 below
– a policy attached to the group customers: Policy 2
– the general Bucket Policy in the Permissions tab of the bucket Policy 3, which includes your AWS account ID. You can find this on top of the billing console.
- From the console you can now customize the link to give to customers along with their username and password. They should be able to upload data to their folder using the web interface which I can then process.
Policy 1:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowListBucketIfSpecificPrefixIsIncludedInRequest",
"Action": [
"s3:ListBucket"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::my-company-bucket"
],
"Condition": {
"StringLike": {
"s3:prefix": [
"customer-A/*"
]
}
}
},
{
"Sid": "AllowUserToReadWriteObjectDataInDevelopmentFolder",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::my-company-bucket/customer-A/*"
]
}
]
}
Policy 2:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowGroupToSeeBucketListAndAlsoAllowGetBucketLocationRequiredForListBucket",
"Action": ["s3:ListAllMyBuckets", "s3:GetBucketLocation"],
"Effect": "Allow",
"Resource": ["arn:aws:s3:::*"]
},
{
"Sid": "AllowRootLevelListingOfCompanyBucket",
"Action": ["s3:ListBucket"],
"Effect": "Allow",
"Resource": ["arn:aws:s3:::my-company-bucket"],
"Condition":{
"StringEquals":{"s3:prefix":[""]}
}
},
{
"Sid": "RequireFolderStyleList",
"Action": ["s3:ListBucket"],
"Effect": "Deny",
"Resource": ["arn:aws:s3:::*"],
"Condition":{
"StringNotEquals":{"s3:delimiter":"/"}
}
},
{
"Sid": "ExplictDenyAccessToPrivateFolderToEveryoneInTheGroup",
"Action": ["s3:*"],
"Effect": "Deny",
"Resource":["arn:aws:s3:::my-company-bucket/other-data/*"]
},
{
"Sid": "DenyListBucketOnPrivateFolder",
"Action": ["s3:ListBucket"],
"Effect": "Deny",
"Resource": ["arn:aws:s3:::*"],
"Condition":{
"StringLike":{"s3:prefix":["other-data/"]}
}
}
]
}
Policy 3:
{
"Version": "2012-10-17",
"Id": "Policy1502460168202",
"Statement": [
{
"Sid": "Stmt1502460165827",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::MY-AWS-ID:root"
},
"Action": "s3:*",
"Resource": "arn:aws:s3:::my-company-bucket/*"
}
]
}
As the largest cloud computing company Amazon Web Services (AWS) offers various options to use compute-power on an “as-needed” basis. You can choose what size and type of machine, what number of machines – and you can choose a price model where you are “bidding” for the resource. This means you might have to wait longer to get it, but you will get an impressive discount! You can choose your machines from the AWS dashboard.
Here is an approach to detect the user’s operating system using a bash script. This might be useful during installation routines of your software. Inspired by 