An unified object storage service. It is used as a staging area to move data into other GCP components like Cloud SQL, BigQuery, Cloud Dataproc.
It is a blob storage with any format.
Rather than storing data on the compute engine (It would be expensive as the compute engine goes away, disk goes away too), store it on Cloud Storage. Cloud Storage is a persistent storage, it is durable, replicated and also made globally available via HTTP URL.
Key Terms
Buckets: Basic containers that hold your data. Everything that you store in Google Cloud Storage must be contained in a bucket. You can use buckets to organize your data and control access to your data, but unlike directories and folders, you cannot nest buckets.
Bucket names: Should be unique as the name of the buckets stored in single Cloud Storage namespace. Also, bucket names can be used with a CNAME redirect, which means they need to conform to DNS naming conventions.
Bucket labels: Bucket labels are key:value metadata pairs that allow you to group your buckets along with other Google Cloud.
Objects: Objects are the individual pieces of data that you store in Google Cloud Storage. Objects have two components: object data and object metadata. The object data component is usually a file that you want to store in Google Cloud Storage. The object metadata component is a collection of name-value pairs that describe various object qualities. There is no limit on the number of objects that you can create in a bucket.
Data Handling
You can set up a transfer service for ingesting data into Cloud Storage from AWS S3 buckets, our computers etc., for once or recurring .
Permissions can be set at project, bucket and object level.
Creating Bucket via GCP Console
Click on “CREATE BUCKET”
Enter the name and select the storage class according to the requirement. Click on “Create”.
Creating Bucket via gsutil Command Line
By installing gcloud sdk and follow the tutorial to initialize to select the project and zone location, then you can execute any command. gsutil will be available.
> gsutil mb gs://simpletext
Above command creates a bucket. Go to Storage dashboard.
If you haven’t mentioned the storage classes it will by default selects multi-region.
Copy some text files to the bucket. If you have a large number of files to transfer you might want to use the gsutil -m option, to perform a parallel (multi-threaded/multi-processing) copy.
> gsutil -m cp test*.txt gs://simpletext
Files are available. You can also make files globally available by checking “Share publicly”. Permissions can also be set.
There are many other commands available for moving contents to one bucket to other bucket, list contents of the bucket etc.
Setting Up Trasnfer Service
On the Cloud Storage dashboard, select “Transfer” and then click on “Create transfer”.
Select the source from which you wanted to copy the files, then “Continue”
You can select the destination bucket to copy the files to. Check the box according to the requirement, then “Continue”.
You can now select either “Run Now” or “Run daily at <time>”. Proceed clicking on “Create”.
