Other options for interacting with the OLRC are available, including the Swift CLI and Swift and S3 APIs. Credentials for these and other tools can be retrieved through Horizon.
Credentials
OpenStack RC File
To interact with the OLRC using the Swift command line, you can download an OpenStack RC file containing environment variables. The variables in this file can also be used to set up other tools like rclone. To download this file:
- Select the project that you want this file for in the upper left corner.
- Click your username in the upper right corner, and select “OpenStack RC File.”
- Save the file to your preferred location.
Alternately, you can go to the API Access page from the “Project” menu in the left sidebar to download this file from the “Download OpenStack RC File” button. This button also provides the option to download an “OpenStack clouds.yaml file,” a configuration file with the necessary information to connect the OLRC to other cloud storage providers.
API Access
The OLRC can also be accessed via API. Details about endpoints and credentials can be found by:
- Selecting the project that you would like to access via API.
- Under the “Project” menu in the left sidebar, selecting “API Access” to view a page listing the Identity and Object Store service endpoints.
- Clicking “View Credentials” to retrieve user details.
With some limitations, the OLRC also supports the AWS S3 API. Separate credentials are required for this option.
Application Credentials
You have the option to create application credentials with defined access rules that can be used in other applications. Credentials will be associated with the project that you are in at the time that they are created. For security purposes, we do not recommend creating these credentials to create additional application credentials or keystone trusts. To create application credentials:
- Set your project to the one that you would like credentials for in the upper left corner.
- Expand the “Identity” menu in the left-side menu and select “Application Credentials.”
- Click “Create Application Credential” in the right corner.
- Complete the form and select “Create Application Credential.”
Swift Command Line
To use the Swift command line, you will need to download your OpenStack RC file from Horizon and install the OpenStack and Swift command-line clients.
Installing the CLIs
Python3 is required to install the OpenStack and Swift clients.
python3 -m pip install python-openstackclient python3 -m pip install python-swiftclient
Setting Environment Variables
Note the location of your OpenStack RC file.
For Mac/Linux users, type in a terminal window:
source <path to your openrc file>
For Windows users, it is not common practice to define environment variables using an environment file on Microsoft Windows. Instead, environment variables are usually defined under Advanced > System Properties. One method for using the above command on Windows is to install Git for Windows. You can then use Git Bash to source the environment variables and to run all CLI commands. For additional information on setting environment variables, please refer to the official OpenStack documentation.
Testing the CLI
To confirm that the clients are installed and your environment variables are set properly, try the following commands. They should return basic information about your account without any errors:
openstack token issue swift auth
Using the CLI
To learn all the ways in which you can interact with the OLRC using the CLI, you can either enter:
openstack --help swift --help
Or refer to the OpenStack Swift CLI manual page for a list of commands for interacting with objects in the OLRC, such as listing and uploading files. Documentation on additional commands for more complex functions such as object versioning, object expiry, and automatically extracting archive files is also available.
S3 API
The OLRC is generally compatible with the AWS S3 API, with some caveats due to differences in features and functionality. All basic operations, like object retrieval, upload, deletion, and, container creation, are fully supported. See OpenStack’s S3/Swift REST API Comparison Matrix for a full list of supported operations. The term “container” that Swift uses to refer to a grouping of objects is equivalent to a “bucket” in the S3 ecosystem. We use these terms interchangeably in this documentation.
Generating S3 Credentials
To use the S3 API with the OLRC, you must first install the OpenStack CLI and generate an access key and a secret key. These keys are always scoped to a single project. Once OpenStack is installed, enter the following command to generate S3 credentials: openstack ec2 credentials create The output will look similar to this:
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------+
| access | abcdef012345678997303245e0eb9a7d |
| links | {'self': 'https://olrc2auth.scholarsportal.info/v3/users/66074d7fa08d48ba827f320396e54452/credentials/OS-EC2/26d8741d6d6e4f6497303245e0eb9a7d'} |
| project_id | abcdef012345678992a2f9ca6d9ab7f0 |
| secret | abcdef01234567899f63edc3e0ed7134 |
| trust_id | None |
| user_id | abcdef0123456789827f320396e54452 |
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------+
Note the values for “access” and “secret” in this table. These keys allow you to use almost all CLI and GUI tools that are S3 compatible. Some CLI tools include AWS CLI with the AWS CLI Endpoint Plugin and s3cmd. Please note that we have not tested these tools exhaustively. To configure the endpoint, provide the following parameters:
- Host: https://olrc2.scholarsportal.info
- Access: The access key provided by the above command
- Secret: The secret key provided by the above command
Some software may require some additional parameters:
- Region: RegionOne
- Signature Version: Either V2 or V4
- Addressing style: Either path style or virtual-host style
Other Tools
Along with the Swift and S3 APIs, Horizon, and DuraCloud, you can also use external tools to manage your content in the OLRC.
RClone
RClone is a command-line tool that can be used to transfer data to the OLRC. One of the major benefits of rclone is its ability to synchronize data (similar to a tool like rsync), meaning that only data that has changed will be uploaded. RClone is our recommended tool for uploading/downloading large volumes of data. To use rclone with the OLRC, first download and install rclone.
In your terminal, set up your endpoint with:
rclone config
Depending on how you installed RClone, you may need to specify the full path to the binary. For instance, on a Windows machine, you might need something like:
%homepath%\Downloads\rclone-v1.57.0-windows-amd64\rclone.exe config
Follow the prompts to set up your configuration, and set your storage type as “OpenStack Swift.” Additional setup values can be found in your OpenRC file. Following the prompts from start to finish, you would type these values in this order, replacing endpoint-name, username, password, domain and project with your setup values.
e/n/d/r/c/s/q> n
name> endpoint-name
Storage> swift
env_auth>
user> username
key> password
auth> https://olrc2auth.scholarsportal.info/v3
user_id>
domain> domain
tenant> project
tenant_id>
tenant_domain> domain
region> RegionOne
Once you have setup your endpoint, try the following command to list all of your containers:
rclone lsd endpoint-name:
Useful commands to start using RClone with the OLRC are copy and sync.
To copy the contents of a local directory to an OLRC container, with real-time updates, for instance:
rclone copy --progress /path/to/files endpoint-name:container-name
To synchronize data between a local location and an OLRC container:
rclone sync /local/path/to/sync/from endpoint-name:container-name/optional/path/to/sync/to
See the rclone swift documentation for additional setup help and commands.
Cyberduck and the duck CLI
Cyberduck is a free software GUI client that supports OpenStack Swift operations, including bulk uploads and downloads. The application is available for Windows and macOS. These instructions have been written using macOS, but the steps are similar in a Windows environment. When uploading large volumes of data with Cyberduck, we strongly recommend verifying the data was uploaded via another method (e.g. with Horizon or the command-line) as some users have had issues with failed uploads when uploading large amounts of data using Cyberduck. To use the Cyberduck GUI:
- Download Cyberduck and follow the installation instructions.
- After you have installed and opened Cyberduck, open a new connection and enter your credentials:
- Connection profile (drop-down menu): OpenStack Swift (Keystone 3)
- Server: olrc2auth.scholarsportal.info
- Port: 443
- Project/Domain/Username: Enter the name of your project, domain (e.g. your institution domain), and your account name, formatted as “project:domain:username”. For example, “general:scholarsportal:kaitlin”. You can find these details in the Horizon dashboard as well
- Password: Enter your password (the same password you use to login to Horizon or use the Swift CLI)
- Click “Connect.”
- Once you have connected, you will see a list of the containers in the project. You can now create or delete containers and upload or download data from your local machine.
If you prefer to use the command line, Cyberduck has a CLI equivalent called duck that provides the same functionalities. It is available on Windows, OS X, and is packaged for both DEB and RPM-based Linux distributions. Once you’ve installed duck, you can use it to perform bulk operations. Your OLRC environment variables will need to be mapped to their command line equivalents. To perform a bulk upload, for instance, you would run the following command:
duck --username OS_TENANT:OS_USERNAME --password PASSWORD --upload swift://olrc2.scholarsportal.info:5000 /container/ path-to-directory-on-local-filesystem/
In this example, note that your container name must be followed by a trailing slash “container/”. If the trailing slash is not included, duck will try to replace your container with the files that you’ve selected for upload. A warning message will appear before any data is overwritten. If you are uploading a directory, the path to the directory must also be followed by a trailing slash to indicate that it is not a file.
S3 Browser
S3 Browser is a Windows client that can be used to upload data to the OLRC via the S3 API.