Short: Backup,restore+manage files on Amazon S3 Author: Michal Ludvig / Alex Carmona Uploader: Alex Carmona Type: comm/tcp Version: 1.1 Architecture: ppc-amigaos >= 4.0.0 URL: http://s3tools.org OVERVIEW: S3Cmd is a Python script and libraries which give you access to Amazon S3 from the Shell. It also provides support for CloudFront and S3 Websites. Most of the online storage and backup services out there use proprietary Windows/OSX clients or require using a browser to upload data. As Amazon provide an API to connect directly to their servers I thought that it would be nice if the Amiga too could have transparent access to their cloud storage service. After exploring the various solutions I settled for S3Cmd as it only needs Python which comes pre-installed with OS4 and I wanted to keep it as simple to install as possible so that anyone else could use it too. With the included S3Cmd script you will be able to easily load/save, share and backup/restore data on the cloud with your Amiga. No more excuses for not having backups ;-) REQUIREMENTS: For S3Cmd to be of any use you'll need an Amazon Web Services (AWS) account. Currently Amazon gives a whole year for free from the date you sign up and their free offer includes a lot more than storage. For example you can create an M1 cloud server instance with 30GB of EBS storage, dedicated IP, root access and the ability to install any Linux version and software you want. See http://aws.amazon.com/free/ for details INSTALLATION: 1) Copy the S3C drawer wherever you like 2) If you don't have a $HOME variable defined (try 'GetEnv HOME' to find out), open the Shell and type: 'MakeDir ENVARC:HOME' 'SetEnv SAVE HOME ENVARC:HOME' You can use any drawer you like instead of ENVARC:HOME, it just seems like the best place for it as $HOME is often used by Linux programs to store settings. 3) In the Shell, type: 'DH1:S3C/S3Cmd --configure' (replace DH1: with your path) It will ask for your AWS keys and whether you want to use GPG, HTTPS and proxies. - If you have GPG installed you could try it otherwise answer 'n' and press Enter when it asks for the GPG passphrase. - I recommend using HTTPS just to be sure your keys aren't sent as plain text. - I haven't tested the GPG and Proxy support so I don't know if they work or not. - If needed you can reconfigure S3Cmd later with '--configure'. USAGE: After installing S3Cmd you will be able to access your S3 storage transparently from the Shell with the S3Cmd command. You don't have to 'CD' to the S3C drawer first, you can call it from anywhere. You can use S3Cmd as-is (see the How-To in the S3C/README and 'S3Cmd --help'), for example: 'S3Cmd mb s3://amigarulez' (Create a bucket named "amigarulez" if available) 'S3Cmd put DH1:myfile s3://amigarulez/newdrawer/' (put a file in a new drawer) 'S3Cmd ls s3://amigarulez/newdrawer/' (list contents of newdrawer) 'S3Cmd get s3://amigarulez/newdrawer/myfile RAM:myfile' (download file to RAM:) 'S3Cmd sync DH1:MyPrecious/ s3://amigarulez/MyPrecious/' (incremental backup) 'S3Cmd sync s3://amigarulez/MyPrecious/ DH1:MyPrecious/' (incremental restore) 'S3Cmd -r del s3://amigarulez/newdrawer' (delete newdrawer recursively) 'S3Cmd rb s3://amigarulez' (remove the amigarulez bucket) or you can create aliases and scripts to have more amiga-like commands operating on a single bucket for common tasks such as: S3List, S3Put, S3Get, S3Delete, S3Info, etc. For example, if you own a bucket called "amigarulez": 'Alias S3 "DH1:S3C/S3Cmd"' 'Alias S3List S3 s3://amigarulez/[]' 'Alias S3Put S3 -r [] s3://amigarulez/' 'Alias S3Get S3 -r s3://amigarulez/[]' 'Alias S3Delete S3 del s3://amigarulez/[]' 'Alias S3DelAll S3 -r del s3://amigarulez/[]' 'Alias S3Info S3 info s3://amigarulez/[]' More complex operations like copying files to a new drawer or moving/renaming objects require multiple arguments which can't be concatenated with Alias so you would have to create small scripts, for example an S:Shell/S3Move script: .key SOURCE/A,DESTINATION/A DH1:S3C/S3Cmd mv s3://amigarulez/ s3://amigarulez/ or a more flexible S3Put script than the S3Put alias above, S:Shell/S3Put: .key SOURCE/A/M,DESTINATION/A DH1:S3C/S3Cmd put -r s3://amigarulez/ or an S3Copy script to duplicate files on the remote side, S:Shell/S3Copy: .key SOURCE/A,DESTINATION/A DH1:S3C/S3Cmd cp s3://amigarulez/ s3://amigarulez/ Finally, a script to automatically make either daily, weekly, or monthly backups of the SYS: partition, incrementally (it's fast as it only copies newer file). (To be useful it should be placed in WBStartup or loaded by some cron job) Lab Top Wait UNTIL 12:00 ; set a time when the machine will be ON with light use If `Date LFormat %e` EQ 1 ; 1st of the month DH1:S3C/S3Cmd --delete-removed sync SYS: s3://amigarulez/backups/monthly/ Else If `Date LFORMAT %w` EQ 1 ; Monday DH1:S3C/S3Cmd --delete-removed sync SYS: s3://amigarulez/backups/weekly/ Else ; daily DH1:S3C/S3Cmd --delete-removed sync SYS: s3://amigarulez/backups/daily/ EndIf Skip Top BACK ; Jump to Top and wait again (most likely until next day) HINTS: Trailing slashes matters, always append a slash "/" after drawer names unless you are trying to delete or get some information on the drawer itself. With 'put' and 'sync' if you specify the source as "RAM:Drawer" instead of "RAM:Drawer/" it will create two drawers "RAM" and "RAM/Drawer" which probably isn't what you want. I recommend using a single bucket and neatly organized drawers as it's easier to have a constant S3 URI for your scripts and aliases. If you want to multiple buckets, you could create an ENV:S3Bucket variable and then put $S3Bucket instead of the bucket name in your scripts, that way you can use 'SetEnv S3Bucket name' to switch buckets without modifying the scripts. With aliases you can't store a reference to a variable so you'd have to use escaped backticks like this: 'Alias S3List S3 ls s3://*`GetEnv S3Bucket*`/[]' If you enable user read permission on drawers or files (setacl) you can share your files by giving people links such as: http://s3.amazonaws.com/amigarulez/photos/myphoto.jpg or http://amigarulez.s3.amazonaws.com/photos/myphoto.jpg Use the 'S3Cmd du' command to keep an eye on how much storage you use, remember that you can only use up to 5GB for free on S3. As you also get plenty of storage on the EC2/EBS side, you could always move stuff from S3 to EBS. If you have Python: on an FFS partition, you can speed it up a great deal by activating caching with 'C:fs_plugin_cache SYS: S=20000' (20MB cache). With ARexx for Python and ProAction from Andy Broad you can quickly create some ReAction GUIs to interact with S3. If someone has the time and patience to make an s3-handler to mount buckets as volumes like FTPMount and NetFS, please do! If you run into issues, ask on the AW forum, that's the one I am most likely to visit regularly. CREDITS: All the hard work was done by Michal Ludvig (S3Cmd), I only added a few lines of code to prevent infinite loops and crashes from the colon in volume names. Enjoy!