Storage classes work best when the frequency of operations on objects in a bucket can be predicted. But what about unpredictable access patterns? Although you can change the storage class of an individual object via a command line (a storage class is a metadata of an object), this could be unprofitable because the object will have to be rewritten, possibly incurring additional operation fees.
For such cases, Google Cloud Storage offers the following options to manage an object’s class assignment:
- Object Lifecycle Management is a feature you can set at a bucket level that allows changing a class of an object without a rewrite operation and the cost that goes with it (although an early deletion fee could be applied). We can set the conditions such as an object’s age and an automatic action that follows when this condition is met – for example, to move an object to a colder class (for example, from Standard to Nearline or from Nearline to Coldline, etc.) or delete it.
- The Autoclass feature is also a bucket-level setting configured during its creation. It automatically migrates infrequently accessed objects to colder classes without retrieval fees and class transition charges.
Figure 8.4 – Because an object is not accessed, the Autoclass feature progressively transitions it to a less expensive storage class
The preceding figure presents how the Autoclass feature changes storage class metadata for a cat.jpg object. First, the original Standard class changes to Nearline after no one has accessed the object for 30 days. Next, after 60 days, the class changes to Coldline. When the object is moved to Coldline, it hasn’t been accessed for 30+60=90 days. Still, if no one accesses it in the next 275 days, its class will change to Archive. But anytime someone accesses cat.jpg, regardless of its storage class, it will return to the Standard class.
Note that in both cases (Object Lifecycle and Autoclass), it is an individual object for which a storage class (its metadata) is changed. Buckets are not modified during this process, so there is no impact on your applications or users accessing data.
Working with buckets and objects
Now that we know how to design Google Cloud Storage buckets for availability and cost efficiency, let’s see how we can upload and manage objects that go into our buckets.
Managing Google Cloud Storage via the Google Cloud console
The Google Cloud console is the easiest way for single-bucket operations and object uploads. You can access the Buckets section by selecting Cloud Storage in the main menu, as shown in the following screenshot:
Figure 8.5 – The location of the Buckets section in the Google Cloud console
To create a bucket, use the CREATE button. This action will open a bucket creation wizard where you can name your bucket (a globally unique name is required), provide a location, and set an object class:
Figure 8.6 – Creating a bucket in the Google Cloud console
There are specific rules to follow when naming a bucket. For example, you can only use lowercase letters, numeric characters, dashes, and underscores. Dots are supported when a bucket contains a domain name but using a domain requires its ownership verification. Also, including “google” as a part of the name is not supported.
We will see the bucket creation process later in this section. Please note that once you create a bucket, you can’t change its name.
As mentioned in the previous section, the only way to change most of the bucket’s parameters is to create a new bucket with desired settings and move contents to this new bucket.
If you want to delete a bucket or multiple buckets with all content, select them in the same view as shown in the preceding screenshot; the DELETE button will appear at the top.
Figure 8.7 – Bucket details view
If you want to download or delete an object or a group of objects from a bucket in the Google Cloud console, use the DOWNLOAD or DELETE options, respectively. Both are available in the Bucket details view, as shown in Figure 8.7.
There are also alternatives to uploading and downloading data from a local workstation. For example, under the TRANSFER DATA option, you will find a wizard to create a transfer job, for example, from object stores belonging to other cloud providers or other Google Cloud Storage buckets.
Figure 8.8 – Ordering Transfer Appliance for uploading large amounts of data
If you are planning to move a large amount of data from an on-premises data center to your Google Cloud Storage bucket, it is important to have a solid strategy in place. For example, if you have a deadline of one month to transfer 100 TB of data over your 1 Gbps bandwidth, it should be easily achievable. However, if you only have 100 Mbps bandwidth available, it could take several months to complete the transfer and you may not meet your deadline. In this scenario, it may be beneficial to consider an offline upload using Transfer Appliance (see Figure 8.8), which can have an end-to-end cycle time of less than a month.
Transfer Appliance is a high-capacity storage device that allows you to securely transfer your data to a Google upload facility, where it will be uploaded to Cloud Storage.
If your data is located in another cloud storage provider, you should consider using the Storage Transfer Service. This service automates the transfer of data between various object and file storage systems, such as Google Cloud Storage, Amazon S3, Azure Storage, or on-premises. It is a reliable and efficient way to transfer large amounts of data without requiring any coding skills.
Using the Google Cloud console is very convenient when learning to work with Google Cloud Storage. Later, when we use it in production, alternative ways to manipulate large amounts of objects are preferred. The following sections will describe how to work with Google Cloud Storage on a larger scale or programmatically.