abstract:darwin:transfer:rclone

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
abstract:darwin:transfer:rclone [2021-03-29 17:43] – [Configuration Steps] anitaabstract:darwin:transfer:rclone [2021-03-29 19:26] (current) – [Configuration Steps] anita
Line 1: Line 1:
 +====== Rclone on DARWIN ======
 +
 +This document describes how to manually configure (set up) Rclone using University of Delaware email account for Google Drive on DARWIN.
 +
 +<note important>Due to firewall rules, the standard documentation for setting up [[https://rclone.org/|Rclone]] on DARWIN might not work for all cloud storage providers. The following directions will show you how to manually set up Rclone and avoid issues with the firewall. </note>
 +
 +===== Rclone manual configuration of Google Drive =====
 +
 +==== Loading Rclone ====
 +First use VALET to load ''rclone'' into your environment
 +
 +<code bash>
 +[traine@login00.darwin ~]$ vpkg_require rclone
 +Adding package `rclone/1.54.1` to your environment
 +[traine@login00.darwin ~]$
 +</code>
 +
 +==== Configuration Steps ====
 +Create a new Rclone remote configuration by typing 'rclone config'. In this example, we are going to calling this new configuration ''darwinGoogleDrive''. You don't have to use this name, you can call your configuration whatever you would like. 
 +
 +<code bash>
 +[traine@login00.darwin ~]$ rclone config
 +2021/03/22 13:19:43 NOTICE: Config file "/home/1200/.config/rclone/rclone.conf" not found - using defaults
 +
 +No remotes found - make a new one
 +n) New remote
 +s) Set configuration password
 +q) Quit config
 +n/s/q> n
 +name> darwinGoogleDrive
 +</code>
 +Next find the "Google Drive" option from the given list of cloud services. Once it is located enter its value, ''15'' or ''drive''. Keep in mind these options can change so please make sure that you read and choose the right option.
 +
 +<code bash>
 +Type of storage to configure.
 +Enter a string value. Press Enter for the default ("").
 +Choose a number from below, or type in your own value
 + 1 / Fichier
 +   \ "fichier"
 + 2 / Alias for an existing remote
 +   \ "alias"
 + 3 / Amazon Drive
 +   \ "amazon cloud drive"
 +
 +   ... Skipping lines...
 +   
 +14 / Google Cloud Storage (this is not Google Drive)
 +   \ "google cloud storage"
 +15 / Google Drive
 +   \ "drive"
 +16 / Google Photos
 +   \ "google photos"
 +17 / Hadoop distributed file system
 +   \ "hdfs"
 +
 +... Skipping more lines ...
 +
 +41 / premiumize.me
 +   \ "premiumizeme"
 +42 / seafile
 +   \ "seafile"
 +Storage> drive
 +</code>
 +
 +The next couple of steps will ask about adding a ''client_id'' and a ''client_secret''. For this example, we select the default option ("") by pressing ''Enter'' for each.
 +
 +<code bash>
 +** See help for drive backend at: https://rclone.org/drive/ **
 +
 +Google Application Client Id
 +Setting your own is recommended.
 +See https://rclone.org/drive/#making-your-own-client-id for how to create your own.
 +If you leave this blank, it will use an internal key which is low performance.
 +Enter a string value. Press Enter for the default ("").
 +client_id>
 +OAuth Client Secret
 +Leave blank normally.
 +Enter a string value. Press Enter for the default ("").
 +client_secret>
 +</code>
 +The next step will give you options to grant Rclone permission to access your files and directories in your Google Drive. We are going to use the default value of ''full access'' by typing ''1'' or ''drive'', but you can read the different options and choose the best option for you. 
 +
 +<code bash>
 +Scope that rclone should use when requesting access from drive.
 +Enter a string value. Press Enter for the default ("").
 +Choose a number from below, or type in your own value
 + 1 / Full access all files, excluding Application Data Folder.
 +   \ "drive"
 + 2 / Read-only access to file metadata and file contents.
 +   \ "drive.readonly"
 +   / Access to files created by rclone only.
 + 3 | These are visible in the drive website.
 +   | File authorization is revoked when the user deauthorizes the app.
 +   \ "drive.file"
 +   / Allows read and write access to the Application Data folder.
 + 4 | This is not visible in the drive website.
 +   \ "drive.appfolder"
 +   / Allows read-only access to file metadata but
 + 5 | does not allow any access to read or download file content.
 +   \ "drive.metadata.readonly"
 +scope> 1
 +</code>
 +
 +For the next two steps, we select the default option ("") by pressing 'Enter' for each.
 +
 +<code bash>
 +ID of the root folder
 +Leave blank normally.
 +
 +Fill in to access "Computers" folders (see docs), or for rclone to use
 +a non root folder as its starting point.
 +
 +Enter a string value. Press Enter for the default ("").
 +root_folder_id>
 +Service Account Credentials JSON file path
 +Leave blank normally.
 +Needed only if you want use SA instead of interactive login.
 +
 +Leading `~` will be expanded in the file name as will environment variables such as `${RCLONE_CONFIG_DIR}`.
 +
 +Enter a string value. Press Enter for the default ("").
 +service_account_file>
 +</code>
 +
 +For the next question about ''advance config'' it is generally not required or needed, so type ''n'' to select 'No' option or read more about ''advance config'' option. 
 +
 +<code bash>
 +Edit advanced config? (y/n)
 +y) Yes
 +n) No (default)
 +y/n> n
 +</code>
 +
 +So far we have been answering the configuration questions mostly with the default options. However the next question is where we will differ from the typical set up process due to firewall permissions restricting ''auto config'' to work correctly, so type ''n'' to select the option working on a remote or headless machine. This option will allow us to safely work around the firewall permission issue by requesting a verification code from our local machine (rather than DARWIN), then we can manually copy and paste the verification code to DARWIN. 
 +
 +<code  bash>
 +Remote config
 +Use auto config?
 + * Say Y if not sure
 + * Say N if you are working on a remote or headless machine
 +y) Yes (default)
 +n) No
 +y/n> n
 +</code>
 +
 +After typing ''n'' and press ''Enter'', a URL will be displayed. This URL will allow you to sign into your University of Delaware Google email account and request a verification code for Rclone. Copy this URL and paste it into the a web browser on your local computer. Sign in and afterwards you will be given your verification code, copy and paste this verification code into the DARWIN shell session.  It will look something like this below:
 +
 +<code bash>
 +Please go to the following link: https://accounts.google.com/o/oauth2/auth?access_type=offline&client_id=**The rest of this link is unique for each user.**
 +Log in and authorize rclone for access
 +Enter verification code> *** ENTER YOU UNIQUE VERIFICATION CODE HERE ***
 +</code>
 +The last stop will ask you if you want to configure this as a "Shared Drive (Team Drive)". For this example, we will type ''n'' and press ''Enter'' to select the No (default) option.
 +<code bash>
 +Configure this as a Shared Drive (Team Drive)?
 +y) Yes
 +n) No (default)
 +y/n> n
 +</code>
 +
 +Now you can review your configuration and make edits if needed. If everything is okay type ''y'' and press ''Enter''.
 +
 +<code bash>
 +--------------------
 +[darwinGoogleDrive]
 +type = drive
 +scope = drive
 +token = {"access_token":**The rest of this is private and unique information for each user.**"}
 +--------------------
 +y) Yes this is OK (default)
 +e) Edit this remote
 +d) Delete this remote
 +y/e/d> y
 +</code>
 +
 +Once everything is configured correctly, Rclone will show you a list of your remote configurations, and give you options to modify, create a new one, or simply exit the Rclone configuration.  At this point, assuming all steps were followed as described, we will simple ''Quit Config'' with the ''q'' option, and press 'Enter'. 
 +
 +<code bash>
 +Name                 Type
 +====                 ====
 +darwinGoogleDrive    drive
 +
 +e) Edit existing remote
 +n) New remote
 +d) Delete remote
 +r) Rename remote
 +c) Copy remote
 +s) Set configuration password
 +q) Quit config
 +e/n/d/r/c/s/q> q
 +</code>
 +
 +==== Testing new Rclone remote configuration ====
 +
 +To make sure the new remote configuration is working correctly, we will use the Rclone ''lsd'' command to list out the root level directories of the remote configuration, ''darwinGoogleDrive'', for our University of Delaware Google Drive on DARWIN.
 +
 +<code bash>
 +[traine@login00.darwin ~]$ rclone lsd darwinGoogleDrive:
 +          -1 2020-11-16 15:43:38        -1 Introduction to Unix Shell on Caviness GDoc
 +          -1 2020-05-05 16:46:11        -1 wifi-map
 +</code>
 +
 +For additional information on Rclone and more options, please visit [[https://rclone.org/|Rclone]]
 +
 +