S3QL User’s Guide

• S3QL
  • Features
  • Development Status
  • Supported Platforms
  • Contributing
• Installation
  • Dependencies
  • Installing S3QL
  • Development Version
  • Running tests requiring remote servers
• Storage Backends
  • Google Storage
  • Amazon S3
  • OpenStack/Swift
  • Rackspace CloudFiles
  • S3 compatible
  • Backblaze B2
  • Local
• Important Rules to Avoid Losing Data
  • Rules in a Nutshell
  • Data Durability
• File System Creation
• Managing File Systems
  • Changing the Passphrase
  • Recovering the master key
  • Restoring Metadata Backups
  • Deleting a file system
  • Upgrading the file system
• Mounting
  • Permission Checking
  • Compression Algorithms
  • Notes about Caching
  • NFS Support
  • Failure Modes
  • Automatic Mounting
• Advanced S3QL Features
  • Snapshotting and Copy-on-Write
  • Getting Statistics
  • Immutable Trees
  • Fast Recursive Removal
  • Runtime Configuration
• Unmounting
• Checking for Errors
  • Checking and repairing internal file system errors
  • Detecting and handling backend data corruption
• Storing Backend Information and Credentials
• Contributed Programs
  • benchmark.py
  • pcp.py
  • s3ql_backup.sh
  • expire_backups.py
  • remove_objects.py
• Tips & Tricks
  • SSH Backend
  • Permanently mounted backup file system
  • Improving copy performance
• Frequently Asked Questions
  • How can I improve the file system throughput?
  • Why does df show 1 TB of free space?
  • Which operating systems are supported?
  • Is there a file size limit?
  • Suppose I want to make a small change in a very large file. Will S3QL download and re-upload the entire file?
  • I don’t quite understand this de-duplication feature…
  • What does the “Transport endpoint not connected” error mean?
  • What does “Backend reports that fs is still mounted elsewhere, aborting” mean?
  • Can I access an S3QL file system on multiple computers simultaneously?
  • What block size should I use?
  • Is there a way to access the file system offline?
• Known Issues
• Manpages
  • The mkfs.s3ql command
  • The s3qladm command
  • The mount.s3ql command
  • The s3qlstat command
  • The s3qlctrl command
  • The s3qlcp command
  • The s3qlrm command
  • The s3qllock command
  • The umount.s3ql command
  • The fsck.s3ql command
  • The s3ql_oauth_client command
  • The s3ql_verify command
  • The pcp command
  • The expire_backups command
• Further Resources / Getting Help
• Implementation Details
  • Metadata Storage
  • Data Storage
  • Data De-Duplication
  • Caching
  • Encryption