Overview of cloud storage systems (2024)

Each cloud storage system is slightly different. Rclone attempts toprovide a unified interface to them, but some underlying differencesshow through.

Features

Here is an overview of the major features of each cloud storage system.

NameHashModTimeCase InsensitiveDuplicate FilesMIME TypeMetadata
1FichierWhirlpool-NoYesR-
Akamai NetstorageMD5, SHA256R/WNoNoR-
Amazon S3 (or S3 compatible)MD5R/WNoNoR/WRWU
Backblaze B2SHA1R/WNoNoR/W-
BoxSHA1R/WYesNo--
Citrix ShareFileMD5R/WYesNo--
DropboxDBHASH ¹RYesNo--
Enterprise File Fabric-R/WYesNoR/W-
FTP-R/W ¹⁰NoNo--
Google Cloud StorageMD5R/WNoNoR/W-
Google DriveMD5, SHA1, SHA256DR/WNoYesR/WDRWU
Google Photos--NoYesR-
HDFS-R/WNoNo--
HiDriveHiDrive ¹²R/WNoNo--
HTTP-RNoNoR-
Internet ArchiveMD5, SHA1, CRC32R/W ¹¹NoNo-RWU
JottacloudMD5R/WYesNoRRW
KoofrMD5-YesNo--
Linkbox-RNoNo--
Mail.ru CloudMailru ⁶R/WYesNo--
Mega--NoYes--
MemoryMD5R/WNoNo--
Microsoft Azure Blob StorageMD5R/WNoNoR/W-
Microsoft Azure Files StorageMD5R/WYesNoR/W-
Microsoft OneDriveQuickXorHash ⁵DR/WYesNoRDRW
OpenDriveMD5R/WYesPartial ⁸--
OpenStack SwiftMD5R/WNoNoR/W-
Oracle Object StorageMD5R/WNoNoR/W-
pCloudMD5, SHA1 ⁷RNoNoW-
PikPakMD5RNoNoR-
premiumize.me--YesNoR-
put.ioCRC-32R/WNoYesR-
Proton DriveSHA1R/WNoNoR-
QingStorMD5- ⁹NoNoR/W-
Quatrix by Maytech-R/WNoNo--
Seafile--NoNo--
SFTPMD5, SHA1 ²DR/WDependsNo--
Sia--NoNo--
SMB-R/WYesNo--
SugarSync--NoNo--
Storj-RNoNo--
Uptobox--NoYes--
WebDAVMD5, SHA1 ³R ⁴DependsNo--
Yandex DiskMD5R/WNoNoR-
Zoho WorkDrive--NoNo--
The local filesystemAllDR/WDependsNo-DRWU

¹ Dropbox supports its own customhash.This is an SHA256 sum of all the 4 MiB block SHA256s.

² SFTP supports checksums if the same login has shell access andmd5sum or sha1sum as well as echo are in the remote's PATH.

³ WebDAV supports hashes when used with Fastmail Files, Owncloud and Nextcloud only.

⁴ WebDAV supports modtimes when used with Fastmail Files, Owncloud and Nextcloud only.

QuickXorHash is Microsoft's own hash.

⁶ Mail.ru uses its own modified SHA1 hash

⁷ pCloud only supports SHA1 (not MD5) in its EU region

⁸ Opendrive does not support creation of duplicate files usingtheir web client interface or other stock clients, but the underlyingstorage platform has been determined to allow duplicate files, and itis possible to create them with rclone. It may be that this is amistake or an unsupported feature.

⁹ QingStor does not support SetModTime for objects bigger than 5 GiB.

¹⁰ FTP supports modtimes for the major FTP servers, and also othersif they advertised required protocol extensions. See thisfor more details.

¹¹ Internet Archive requires option wait_archive to be set to a non-zero valuefor full modtime support.

¹² HiDrive supports its own customhash.It combines SHA1 sums for each 4 KiB block hierarchically to a singletop-level sum.

Hash

The cloud storage system supports various hash types of the objects.The hashes are used when transferring data as an integrity check andcan be specifically used with the --checksum flag in syncs and inthe check command.

To use the verify checksums when transferring between cloud storagesystems they must support a common hash type.

ModTime

Almost all cloud storage systems store some sort of timestampon objects, but several of them not something that is appropriateto use for syncing. E.g. some backends will only write a timestampthat represents the time of the upload. To be relevant for syncingit should be able to store the modification time of the sourceobject. If this is not the case, rclone will only check the filesize by default, though can be configured to check the file hash(with the --checksum flag). Ideally it should also be possible tochange the timestamp of an existing file without having to re-upload it.

KeyExplanation
-ModTimes not supported - times likely the upload time
RModTimes supported on files but can't be changed without re-upload
R/WRead and Write ModTimes fully supported on files
DRModTimes supported on files and directories but can't be changed without re-upload
DR/WRead and Write ModTimes fully supported on files and directories

Storage systems with a - in the ModTime column, means themodification read on objects is not the modification time of thefile when uploaded. It is most likely the time the file was uploaded,or possibly something else (like the time the picture was taken inGoogle Photos).

Storage systems with a R (for read-only) in the ModTime column,means the it keeps modification times on objects, and updates themwhen uploading objects, but it does not support changing only themodification time (SetModTime operation) without re-uploading,possibly not even without deleting existing first. Some operationsin rclone, such as copy and sync commands, will automaticallycheck for SetModTime support and re-upload if necessary to keepthe modification times in sync. Other commands will not workwithout SetModTime support, e.g. touch command on an existingfile will fail, and changes to modification time only on a filesin a mount will be silently ignored.

Storage systems with R/W (for read/write) in the ModTime column,means they do also support modtime-only operations.

Storage systems with D in the ModTime column means that thefollowing symbols apply to directories as well as files.

Case Insensitive

If a cloud storage systems is case sensitive then it is possible tohave two files which differ only in case, e.g. file.txt andFILE.txt. If a cloud storage system is case insensitive then thatisn't possible.

This can cause problems when syncing between a case insensitivesystem and a case sensitive system. The symptom of this is that nomatter how many times you run the sync it never completes fully.

The local filesystem and SFTP may or may not be case sensitivedepending on OS.

  • Windows - usually case insensitive, though case is preserved
  • OSX - usually case insensitive, though it is possible to format case sensitive
  • Linux - usually case sensitive, but there are case insensitive file systems (e.g. FAT formatted USB keys)

Most of the time this doesn't cause any problems as people tend toavoid files whose name differs only by case even on case sensitivesystems.

Duplicate files

If a cloud storage system allows duplicate files then it can have twoobjects with the same name.

This confuses rclone greatly when syncing - use the rclone dedupecommand to rename or remove duplicates.

Restricted filenames

Some cloud storage systems might have restrictions on the charactersthat are usable in file or directory names.When rclone detects such a name during a file upload, it willtransparently replace the restricted characters with similar lookingUnicode characters. To handle the different sets of restricted charactersfor different backends, rclone uses something it calls encoding.

This process is designed to avoid ambiguous file names as much aspossible and allow to move files between many cloud storage systemstransparently.

The name shown by rclone to the user or during log output will onlycontain a minimal set of replaced charactersto ensure correct formatting and not necessarily the actual name usedon the cloud storage.

This transformation is reversed when downloading a file or parsingrclone arguments. For example, when uploading a file named my file?.txtto Onedrive, it will be displayed as my file?.txt on the console, butstored as my file?.txt to Onedrive (the ? gets replaced by the similarlooking character, the so-called "fullwidth question mark").The reverse transformation allows to read a file unusual/name.txtfrom Google Drive, by passing the name unusual/name.txt on the command line(the / needs to be replaced by the similar looking character).

Caveats

The filename encoding system works well in most cases, at leastwhere file names are written in English or similar languages.You might not even notice it: It just works. In some cases it maylead to issues, though. E.g. when file names are written in Chinese,or Japanese, where it is always the Unicode fullwidth variants of thepunctuation marks that are used.

On Windows, the characters :, * and ? are examples of restrictedcharacters. If these are used in filenames on a remote that supports it,Rclone will transparently convert them to their fullwidth Unicodevariants , and when downloading to Windows, and back againwhen uploading. This way files with names that are not allowed on Windowscan still be stored.

However, if you have files on your Windows system originally with these sameUnicode characters in their names, they will be included in the same conversionprocess. E.g. if you create a file in your Windows filesystem with nameTest:1.jpg, where is the Unicode fullwidth colon symbol, and userclone to upload it to Google Drive, which supports regular : (halfwidthquestion mark), rclone will replace the fullwidth : with thehalfwidth : and store the file as Test:1.jpg in Google Drive. Sinceboth Windows and Google Drive allows the name Test:1.jpg, it wouldprobably be better if rclone just kept the name as is in this case.

With the opposite situation; if you have a file named Test:1.jpg,in your Google Drive, e.g. uploaded from a Linux system where : is validin file names. Then later use rclone to copy this file to your Windowscomputer you will notice that on your local disk it gets renamedto Test:1.jpg. The original filename is not legal on Windows, due tothe :, and rclone therefore renames it to make the copy possible.That is all good. However, this can also lead to an issue: If you alreadyhad a different file named Test:1.jpg on Windows, and then use rcloneto copy either way. Rclone will then treat the file originally namedTest:1.jpg on Google Drive and the file originally named Test:1.jpgon Windows as the same file, and replace the contents from one with the other.

Its virtually impossible to handle all cases like these correctly in allsituations, but by customizing the encoding option, changing theset of characters that rclone should convert, you should be able tocreate a configuration that works well for your specific situation.See also the example below.

(Windows was used as an example of a file system with many restrictedcharacters, and Google drive a storage system with few.)

Default restricted characters

The table below shows the characters that are replaced by default.

When a replacement character is found in a filename, this characterwill be escaped with the character to avoid ambiguous file names.(e.g. a file named ␀.txt would shown as ‛␀.txt)

Each cloud storage backend can use a different set of characters,which will be specified in the documentation for each backend.

CharacterValueReplacement
NUL0x00
SOH0x01
STX0x02
ETX0x03
EOT0x04
ENQ0x05
ACK0x06
BEL0x07
BS0x08
HT0x09
LF0x0A
VT0x0B
FF0x0C
CR0x0D
SO0x0E
SI0x0F
DLE0x10
DC10x11
DC20x12
DC30x13
DC40x14
NAK0x15
SYN0x16
ETB0x17
CAN0x18
EM0x19
SUB0x1A
ESC0x1B
FS0x1C
GS0x1D
RS0x1E
US0x1F
/0x2F
DEL0x7F

The default encoding will also encode these file names as they areproblematic with many cloud storage systems.

File nameReplacement
.
....

Invalid UTF-8 bytes

Some backends only support a sequence of well formed UTF-8 bytesas file or directory names.

In this case all invalid UTF-8 bytes will be replaced with a quotedrepresentation of the byte value to allow uploading a file to such abackend. For example, the invalid byte 0xFE will be encoded as ‛FE.

A common source of invalid UTF-8 bytes are local filesystems, that storenames in a different encoding than UTF-8 or UTF-16, like latin1. See thelocal filenames section for details.

Encoding option

Most backends have an encoding option, specified as a flag--backend-encoding where backend is the name of the backend, or asa config parameter encoding (you'll need to select the Advancedconfig in rclone config to see it).

This will have default value which encodes and decodes characters insuch a way as to preserve the maximum number of characters (seeabove).

However this can be incorrect in some scenarios, for example if youhave a Windows file system with Unicode fullwidth characters, or , that you want to remain as those characters on theremote rather than being translated to regular (halfwidth) *, ? and :.

The --backend-encoding flags allow you to change that. You candisable the encoding completely with --backend-encoding None or setencoding = None in the config file.

Encoding takes a comma separated list of encodings. You can see thelist of all possible values by passing an invalid value to thisflag, e.g. --local-encoding "help". The command rclone help flags encodingwill show you the defaults for the backends.

EncodingCharactersEncoded as
Asterisk*
BackQuote`
BackSlash\
Colon:
CrLfCR 0x0D, LF 0x0A,
CtlAll control characters 0x00-0x1F␀␁␂␃␄␅␆␇␈␉␊␋␌␍␎␏␐␑␒␓␔␕␖␗␘␙␚␛␜␝␞␟
DelDEL 0x7F
Dollar$
Dot. or .. as entire string, ..
DoubleQuote"
Hash#
InvalidUtf8An invalid UTF-8 character (e.g. latin1)
LeftCrLfHtVtCR 0x0D, LF 0x0A, HT 0x09, VT 0x0B on the left of a string, , ,
LeftPeriod. on the left of a string.
LeftSpaceSPACE on the left of a string
LeftTilde~ on the left of a string
LtGt<, >,
NoneNo characters are encoded
Percent%
Pipe|
Question?
RightCrLfHtVtCR 0x0D, LF 0x0A, HT 0x09, VT 0x0B on the right of a string, , ,
RightPeriod. on the right of a string.
RightSpaceSPACE on the right of a string
Semicolon;
SingleQuote'
Slash/
SquareBracket[, ],
Encoding example: FTP

To take a specific example, the FTP backend's default encoding is

--ftp-encoding "Slash,Del,Ctl,RightSpace,Dot"

However, let's say the FTP server is running on Windows and can't haveany of the invalid Windows characters in file names. You are backingup Linux servers to this FTP server which do have those characters infile names. So you would add the Windows set which are

Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot

to the existing ones, giving:

Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot,Del,RightSpace

This can be specified using the --ftp-encoding flag or using an encoding parameter in the config file.

Encoding example: Windows

As a nother example, take a Windows system where there is a file withname Test:1.jpg, where is the Unicode fullwidth colon symbol.When using rclone to copy this to a remote which supports :,the regular (halfwidth) colon (such as Google Drive), you will noticethat the file gets renamed to Test:1.jpg.

To avoid this you can change the set of characters rclone should convertfor the local filesystem, using command-line argument --local-encoding.Rclone's default behavior on Windows corresponds to

--local-encoding "Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot"

If you want to use fullwidth characters , and in your filenameswithout rclone changing them when uploading to a remote, then set the same asthe default value but without Colon,Question,Asterisk:

--local-encoding "Slash,LtGt,DoubleQuote,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot"

Alternatively, you can disable the conversion of any characters with --local-encoding None.

Instead of using command-line argument --local-encoding, you may also set itas environment variable RCLONE_LOCAL_ENCODING,or configure a remote of type local in your config,and set the encoding option there.

The risk by doing this is that if you have a filename with the regular (halfwidth):, * and ? in your cloud storage, and you try to downloadit to your Windows filesystem, this will fail. These characters are notvalid in filenames on Windows, and you have told rclone not to work aroundthis by converting them to valid fullwidth variants.

MIME Type

MIME types (also known as media types) classify types of documentsusing a simple text classification, e.g. text/html orapplication/pdf.

Some cloud storage systems support reading (R) the MIME type ofobjects and some support writing (W) the MIME type of objects.

The MIME type can be important if you are serving files directly toHTTP from the storage system.

If you are copying from a remote which supports reading (R) to aremote which supports writing (W) then rclone will preserve the MIMEtypes. Otherwise they will be guessed from the extension, or theremote itself may assign the MIME type.

Metadata

Backends may or may support reading or writing metadata. They maysupport reading and writing system metadata (metadata intrinsic tothat backend) and/or user metadata (general purpose metadata).

The levels of metadata support are

KeyExplanation
RRead only System Metadata on files only
RWRead and write System Metadata on files only
RWURead and write System Metadata and read and write User Metadata on files only
DRRead only System Metadata on files and directories
DRWRead and write System Metadata on files and directories
DRWURead and write System Metadata and read and write User Metadata on files and directories

See the metadata docs for more info.

Optional Features

All rclone remotes support a base command set. Other features dependupon backend-specific capabilities.

NamePurgeCopyMoveDirMoveCleanUpListRStreamUploadMultithreadUploadLinkSharingAboutEmptyDir
1FichierNoYesYesNoNoNoNoNoYesNoYes
Akamai NetstorageYesNoNoNoNoYesYesNoNoNoYes
Amazon S3 (or S3 compatible)NoYesNoNoYesYesYesYesYesNoNo
Backblaze B2NoYesNoNoYesYesYesYesYesNoNo
BoxYesYesYesYesYesNoYesNoYesYesYes
Citrix ShareFileYesYesYesYesNoNoNoNoNoNoYes
DropboxYesYesYesYesNoNoYesNoYesYesYes
Enterprise File FabricYesYesYesYesYesNoNoNoNoNoYes
FTPNoNoYesYesNoNoYesNoNoNoYes
Google Cloud StorageYesYesNoNoNoYesYesNoNoNoNo
Google DriveYesYesYesYesYesYesYesNoYesYesYes
Google PhotosNoNoNoNoNoNoNoNoNoNoNo
HDFSYesNoYesYesNoNoYesNoNoYesYes
HiDriveYesYesYesYesNoNoYesNoNoNoYes
HTTPNoNoNoNoNoNoNoNoNoNoYes
ImageKitYesYesYesNoNoNoNoNoNoNoYes
Internet ArchiveNoYesNoNoYesYesNoNoYesYesNo
JottacloudYesYesYesYesYesYesNoNoYesYesYes
KoofrYesYesYesYesNoNoYesNoYesYesYes
Mail.ru CloudYesYesYesYesYesNoNoNoYesYesYes
MegaYesNoYesYesYesNoNoNoYesYesYes
MemoryNoYesNoNoNoYesYesNoNoNoNo
Microsoft Azure Blob StorageYesYesNoNoNoYesYesYesNoNoNo
Microsoft Azure Files StorageNoYesYesYesNoNoYesYesNoYesYes
Microsoft OneDriveYesYesYesYesYesYes ⁵NoNoYesYesYes
OpenDriveYesYesYesYesNoNoNoNoNoNoYes
OpenStack SwiftYes ¹YesNoNoNoYesYesNoNoYesNo
Oracle Object StorageNoYesNoNoYesYesYesYesNoNoNo
pCloudYesYesYesYesYesNoNoNoYesYesYes
PikPakYesYesYesYesYesNoNoNoYesYesYes
premiumize.meYesNoYesYesNoNoNoNoYesYesYes
put.ioYesNoYesYesYesNoYesNoNoYesYes
Proton DriveYesNoYesYesYesNoNoNoNoYesYes
QingStorNoYesNoNoYesYesNoNoNoNoNo
Quatrix by MaytechYesYesYesYesNoNoNoNoNoYesYes
SeafileYesYesYesYesYesYesYesNoYesYesYes
SFTPNoYes ⁴YesYesNoNoYesNoNoYesYes
SiaNoNoNoNoNoNoYesNoNoNoYes
SMBNoNoYesYesNoNoYesYesNoNoYes
SugarSyncYesYesYesYesNoNoYesNoYesNoYes
StorjYes ²YesYesNoNoYesYesNoYesNoNo
UptoboxNoYesYesYesNoNoNoNoNoNoNo
WebDAVYesYesYesYesNoNoYes ³NoNoYesYes
Yandex DiskYesYesYesYesYesNoYesNoYesYesYes
Zoho WorkDriveYesYesYesYesNoNoNoNoNoYesYes
The local filesystemNoNoYesYesNoNoYesYesNoYesYes

¹ Note Swift implements this in order to delete directory markers butit doesn't actually have a quicker way of deleting files other thandeleting them individually.

² Storj implements this efficiently only for entire buckets. Ifpurging a directory inside a bucket, files are deleted individually.

³ StreamUpload is not supported with Nextcloud

⁴ Use the --sftp-copy-is-hardlink flag to enable.

⁵ Use the --onedrive-delta flag to enable.

Purge

This deletes a directory quicker than just deleting all the files inthe directory.

Copy

Used when copying an object to and from the same remote. This knownas a server-side copy so you can copy a file without downloading itand uploading it again. It is used if you use rclone copy orrclone move if the remote doesn't support Move directly.

If the server doesn't support Copy directly then for copy operationsthe file is downloaded then re-uploaded.

Move

Used when moving/renaming an object on the same remote. This is knownas a server-side move of a file. This is used in rclone move if theserver doesn't support DirMove.

If the server isn't capable of Move then rclone simulates it withCopy then delete. If the server doesn't support Copy then rclonewill download the file and re-upload it.

DirMove

This is used to implement rclone move to move a directory ifpossible. If it isn't then it will use Move on each file (whichfalls back to Copy then download and upload - see Move section).

CleanUp

This is used for emptying the trash for a remote by rclone cleanup.

If the server can't do CleanUp then rclone cleanup will return anerror.

‡‡ Note that while Box implements this it has to delete every fileindividually so it will be slower than emptying the trash via the WebUI

ListR

The remote supports a recursive list to list all the contents beneatha directory quickly. This enables the --fast-list flag to work.See the rclone docs for more details.

StreamUpload

Some remotes allow files to be uploaded without knowing the file sizein advance. This allows certain operations to work without spooling thefile to local disk first, e.g. rclone rcat.

MultithreadUpload

Some remotes allow transfers to the remote to be sent as chunks inparallel. If this is supported then rclone will use multi-threadcopying to transfer files much faster.

LinkSharing

Sets the necessary permissions on a file or folder and prints a linkthat allows others to access them, even if they don't have an accounton the particular cloud provider.

About

Rclone about prints quota information for a remote. Typical outputincludes bytes used, free, quota and in trash.

If a remote lacks about capability rclone about remote:returnsan error.

Backends without about capability cannot determine free space for anrclone mount, or use policy mfs (most free space) as a member of anrclone union remote.

See rclone about command

EmptyDir

The remote supports empty directories. See Limitationsfor details. Most Object/Bucket-based remotes do not support this.

Overview of cloud storage systems (2024)

References

Top Articles
Latest Posts
Article information

Author: Trent Wehner

Last Updated:

Views: 5661

Rating: 4.6 / 5 (56 voted)

Reviews: 87% of readers found this page helpful

Author information

Name: Trent Wehner

Birthday: 1993-03-14

Address: 872 Kevin Squares, New Codyville, AK 01785-0416

Phone: +18698800304764

Job: Senior Farming Developer

Hobby: Paintball, Calligraphy, Hunting, Flying disc, Lapidary, Rafting, Inline skating

Introduction: My name is Trent Wehner, I am a talented, brainy, zealous, light, funny, gleaming, attractive person who loves writing and wants to share my knowledge and understanding with you.