Formerly known as ICGC Storage and currently used as the storage and transfer system for ICGC cloud based projects against S3 and Azure backends.
+
+
+
-
+Genomics data volume and velocity have increased dramatically, rendering on-premise storage insufficient and demanding specialized software tools to manage data in the cloud. [Score](https://www.overture.bio/documentation/score/) addresses this by facilitating the transfer and storage of genomics data to and from a cloud network.
-[](https://www.codacy.com/app/overture-stack/score?utm_source=github.com&utm_medium=referral&utm_content=overture-stack/score&utm_campaign=Badge_Grade)
-[](https://score-docs.readthedocs.io/en/latest/?badge=latest)
-[](http://slack.overture.bio)
+
-## Documentation
-
-Explore documentation with the Score [Read the Docs](https://score-docs.readthedocs.io/en/develop/introduction.html).
-
-## Build
-
-To compile, test and package the system, execute the following from the root of the repository:
-
-```shell
-mvn
-```
-
-## Run
-
-See module-specific documentation below.
+
-## Modules
-Top level system modules:
+>
+>
+>
+>
+>
+> *Score is a core component within the [Overture](https://www.overture.bio/) research software ecosystem. Overture is a toolkit of modular software components made to build into scalable genomics data management systems. See our [related products](#related-products) for more information on what Overture can offer.*
+>
+>
+
-- [Core](score-core/README.md)
-- [Client](score-client/README.md)
-- [File System](score-fs/README.md)
-- [Server](score-server/README.md)
-- [Test](score-test/README.md)
+## Technical Specifications
-## Development
-Several `make` targets are provided for locally deploying dependent services using docker.
-By using this, the developer will be able to replicate a live environment for score-server and score-client.
-It allows the user to develop locally, and test uploads/downloads in an isolated environment.
+- Written in JAVA
+- Supports AWS S3, Azure, Google Cloud, Openstack with Ceph, Minio and all other S3-compliant cloud storage solutions
+- Built-in [Samtools](http://www.htslib.org/) functionality including BAM and CRAM file slicing by genomic region
+- ACL security using [OAuth 2.0](https://oauth.net/2/) and scopes based on study codes
+- Multipart Uploads and Downloads
+- REST API with [Swagger UI](https://swagger.io/tools/swagger-ui/)
+- [MD5sum](https://www.intel.com/content/www/us/en/support/programmable/articles/000078103.html) validation
-There are 2 modes:
-
-### 1. Development Mode
-The purpose of this mode is to decrease the wait time between building and testing against dependent services.
-This mode will run a `mvn package` if the `*-dist.tar.gz` files are missing, and copy them into a container for them to be run.
-This method allows for fast developement, since the `mvn package` step is handled on the **Docker host**.
-In addition, the debug ports `5005` and `5006` are exposed for both `score-client` and `score-server`, respectively, allowing developers to debug the docker containers.
-This mode can be enabled using the `DEMO_MODE=0` override. This is the default behaviour if the variable `DEMO_MODE` is not defined.
-
-#### Debugging the score-client with IntelliJ
-Since the JVM debug port is exposed by the `score-client` docker container, IntelliJ can **remotely debug** a running docker container.
-To do this, a **docker image run profile** must be created with the configuration outputted by the `make intellij-score-client-config` command, which will output a basic upload command, however it can be modified to be any score-client command.
-Then, a **remote debug profile** must be created, with the following config:
-
-```
-Host: localhost
-Port: 5005
-Use module classpath: score-client
-```
-and in the `Before launch: Activate tool window` section, click the `+` sign, and select `Launch docker before debug`.
-Then ensure the `Docker configuration` field is set to the name of the previously created **docker image run profile** and that `Custom Options` is set to `-p 5005:5005`. In order for the debugger to bind to the debug port in time,
-a delay needs to be introduced after starting the container. To do this, click the `+` sign again, and select `Launch docker before debug`, and select `Run External Tool` and a window will pop-up. Input the following:
-
-```
-Name: Sleep for 5 seconds
-Program: /usr/bin/sleep
-Arguments: 5
-```
-and click `OK`.
-
-Finally, start debugging by simply running the **remote debug profile** and it will call the **docker image run profile** before launch.
-
-#### Debugging the score-server with IntelliJ
-Since the `score-server` is a server and exposes the 5006 debug port, configuration is much easier. First, start the server with `make clean start-score-server`. Then, create a **remote debug profile** in Intellij with the following configuration:
-```
-Host: localhost
-Port: 5006
-Use module classpath: score-server
-```
-and then run it in debug mode.
+## Documentation
+- :construction: Developer documentation, including instructions for running Score from source can be found in the [Wiki](https://github.com/overture-stack/score/wiki) :construction:
+- For user documentation, including installation, configuration and usage guides, see the Overture websites [Score documentation page](https://www.overture.bio/documentation/score/)
+## Support & Contributions
-### 2. Demo Mode
-The purpose of this mode is to demo the current `score-server` and `score-client` code by building it in **inside the Docker image**,
-as opposed to the **Docker host** as is done in Development mode and then running the containers.
-This mode will not run `mvn package` on the Docker host, but instead inside the Docker container.
-This method is very slow, since maven will download dependencies every time a build is triggered, however creates a completely isolated environment for testing.
-This mode can be enabled using the `DEMO_MODE=1` make variable override. For example, to start the score-server, the following command would be run:
+- Filing an [issue](https://github.com/overture-stack/score/issues)
+- Making a [contribution](CONTRIBUTING.md)
+- Connect with us on [Slack](http://slack.overture.bio)
+- Add or Upvote a [feature request](https://github.com/overture-stack/score/issues?q=is%3Aopen+is%3Aissue+label%3Anew-feature+sort%3Areactions-%2B1-desc)
-```bash
-make start-score-server DEMO_MODE=1
-```
+## Related Software
-For more information on the different targets, run `make help` or read the comments above each target for a description
+
+
+
+Score commonly works in tandem with our metadata service, [Song](https://github.com/overture-stack/SONG). While Score handles object storage and file transfer, Song validates and tracks all the associated file metadata.
-## Changes
+All our core microservices are included in the Overture **Data Management System** (DMS). Built from our core collection of microservices, the DMS offers turnkey installation, configuration, and deployment of Overture software. For more information on the DMS, read our [DMS documentation](https://www.overture.bio/documentation/dms/).
-Change log for the user-facing system modules may be found in [CHANGES.md](CHANGES.md).
+See the links below for information on our other research software tools:
-## License
+
-Copyright and license information may be found in [LICENSE.md](LICENSE.md).
+|Software|Description|
+|---|---|
+|[Ego](https://www.overture.bio/products/ego/)|An authorization and user management service|
+|[Ego UI](https://www.overture.bio/products/ego-ui/)|A UI for managing Ego authentication and authorization services|
+|[Score](https://www.overture.bio/products/score/)| Transfer data to and from any cloud-based storage system|
+|[Song](https://www.overture.bio/products/song/)|Catalog and manage metadata associated to file data spread across cloud storage systems|
+|[Maestro](https://www.overture.bio/products/maestro/)|Organizing your distributed data into a centralized Elasticsearch index|
+|[Arranger](https://www.overture.bio/products/arranger/)|A search API with reusable UI components that build into configurable and functional data portals|
+|[DMS-UI](https://github.com/overture-stack/dms-ui)|A simple web browser UI that integrates Ego and Arranger|
\ No newline at end of file
diff --git a/code_of_conduct.md b/code_of_conduct.md
new file mode 100755
index 00000000..45d257b2
--- /dev/null
+++ b/code_of_conduct.md
@@ -0,0 +1,133 @@
+
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our
+community a harassment-free experience for everyone, regardless of age, body
+size, visible or invisible disability, ethnicity, sex characteristics, gender
+identity and expression, level of experience, education, socio-economic status,
+nationality, personal appearance, race, caste, color, religion, or sexual
+identity and orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming,
+diverse, inclusive, and healthy community.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment for our
+community include:
+
+* Demonstrating empathy and kindness toward other people
+* Being respectful of differing opinions, viewpoints, and experiences
+* Giving and gracefully accepting constructive feedback
+* Accepting responsibility and apologizing to those affected by our mistakes,
+ and learning from the experience
+* Focusing on what is best not just for us as individuals, but for the overall
+ community
+
+Examples of unacceptable behavior include:
+
+* The use of sexualized language or imagery, and sexual attention or advances of
+ any kind
+* Trolling, insulting or derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or email address,
+ without their explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+ professional setting
+
+## Enforcement Responsibilities
+
+Community leaders are responsible for clarifying and enforcing our standards of
+acceptable behavior and will take appropriate and fair corrective action in
+response to any behavior that they deem inappropriate, threatening, offensive,
+or harmful.
+
+Community leaders have the right and responsibility to remove, edit, or reject
+comments, commits, code, wiki edits, issues, and other contributions that are
+not aligned to this Code of Conduct, and will communicate reasons for moderation
+decisions when appropriate.
+
+## Scope
+
+This Code of Conduct applies within all community spaces, and also applies when
+an individual is officially representing the community in public spaces.
+Examples of representing our community include using an official e-mail address,
+posting via an official social media account, or acting as an appointed
+representative at an online or offline event.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported to the community leaders responsible for enforcement at
+[INSERT CONTACT METHOD].
+All complaints will be reviewed and investigated promptly and fairly.
+
+All community leaders are obligated to respect the privacy and security of the
+reporter of any incident.
+
+## Enforcement Guidelines
+
+Community leaders will follow these Community Impact Guidelines in determining
+the consequences for any action they deem in violation of this Code of Conduct:
+
+### 1. Correction
+
+**Community Impact**: Use of inappropriate language or other behavior deemed
+unprofessional or unwelcome in the community.
+
+**Consequence**: A private, written warning from community leaders, providing
+clarity around the nature of the violation and an explanation of why the
+behavior was inappropriate. A public apology may be requested.
+
+### 2. Warning
+
+**Community Impact**: A violation through a single incident or series of
+actions.
+
+**Consequence**: A warning with consequences for continued behavior. No
+interaction with the people involved, including unsolicited interaction with
+those enforcing the Code of Conduct, for a specified period of time. This
+includes avoiding interactions in community spaces as well as external channels
+like social media. Violating these terms may lead to a temporary or permanent
+ban.
+
+### 3. Temporary Ban
+
+**Community Impact**: A serious violation of community standards, including
+sustained inappropriate behavior.
+
+**Consequence**: A temporary ban from any sort of interaction or public
+communication with the community for a specified period of time. No public or
+private interaction with the people involved, including unsolicited interaction
+with those enforcing the Code of Conduct, is allowed during this period.
+Violating these terms may lead to a permanent ban.
+
+### 4. Permanent Ban
+
+**Community Impact**: Demonstrating a pattern of violation of community
+standards, including sustained inappropriate behavior, harassment of an
+individual, or aggression toward or disparagement of classes of individuals.
+
+**Consequence**: A permanent ban from any sort of public interaction within the
+community.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+version 2.1, available at
+[https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].
+
+Community Impact Guidelines were inspired by
+[Mozilla's code of conduct enforcement ladder][Mozilla CoC].
+
+For answers to common questions about this code of conduct, see the FAQ at
+[https://www.contributor-covenant.org/faq][FAQ]. Translations are available at
+[https://www.contributor-covenant.org/translations][translations].
+
+[homepage]: https://www.contributor-covenant.org
+[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
+[Mozilla CoC]: https://github.com/mozilla/diversity
+[FAQ]: https://www.contributor-covenant.org/faq
+[translations]: https://www.contributor-covenant.org/translations
diff --git a/docker/ego-init/init.sql b/docker/ego-init/init.sql
index c541a2a8..d2ad8b74 100644
--- a/docker/ego-init/init.sql
+++ b/docker/ego-init/init.sql
@@ -372,9 +372,9 @@ ed5149c4-d8c3-46f8-ab01-b903f82b5fe3 \N score.TEST-CA
--
COPY public.token (id, name, owner, issuedate, isrevoked, description, expirydate) FROM stdin;
-f7d708ef-41f8-493f-ad8e-cb0ac97b0688 f69b726d-d40f-4261-b105-1ec7e6bf04d5 c6608c3e-1181-4957-99c4-094493391096 2019-11-20 20:52:08.247 f \N 2020-11-19 20:52:08.247
-fafaac34-6b01-47ef-9ae6-6d8cb30af5ca fd0c6c40-254b-4a5f-82e7-cf21a380ccb3 c6608c3e-1181-4957-99c4-094493391096 2019-11-20 20:55:56.186 f \N 2020-11-19 20:55:56.186
-7df26ca6-801f-4302-a318-6f766d759b1d 1f070fb0-0ee4-4815-8097-b5b065c661cc c6608c3e-1181-4957-99c4-094493391096 2019-11-20 20:57:38.345 f \N 2020-11-19 20:57:38.345
+f7d708ef-41f8-493f-ad8e-cb0ac97b0688 f69b726d-d40f-4261-b105-1ec7e6bf04d5 c6608c3e-1181-4957-99c4-094493391096 2023-11-02 20:52:08.247 f \N 2030-11-02 20:52:08.247
+fafaac34-6b01-47ef-9ae6-6d8cb30af5ca fd0c6c40-254b-4a5f-82e7-cf21a380ccb3 c6608c3e-1181-4957-99c4-094493391096 2023-11-02 20:55:56.186 f \N 2030-11-02 20:55:56.186
+7df26ca6-801f-4302-a318-6f766d759b1d 1f070fb0-0ee4-4815-8097-b5b065c661cc c6608c3e-1181-4957-99c4-094493391096 2023-11-02 20:57:38.345 f \N 2030-11-02 20:57:38.345
\.
diff --git a/icon-score.png b/icon-score.png
new file mode 100755
index 00000000..0ba74bdb
Binary files /dev/null and b/icon-score.png differ
diff --git a/ov-logo.png b/ov-logo.png
new file mode 100755
index 00000000..51fda130
Binary files /dev/null and b/ov-logo.png differ
diff --git a/pom.xml b/pom.xml
index 05104cd8..7357819d 100644
--- a/pom.xml
+++ b/pom.xml
@@ -20,7 +20,7 @@ ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF S
bio.overturescore
- 5.9.0
+ 5.10.0pom${project.artifactId}${project.name}
diff --git a/score-client/README.md b/score-client/README.md
index 3e5d1590..fedec157 100644
--- a/score-client/README.md
+++ b/score-client/README.md
@@ -78,5 +78,10 @@ To enable logging of request bodies and headers, append the following to the com
`--logging.level.org.apache.http=DEBUG`
## Azure
-For azure support, use the `azure` profile. Block sizes for uploads
+
+Score-client dynamically loads storage profile specific implementations (currently S3 or Azure) based on the profile score-server is running on.
+It does this by calling an endpoint on score-server to fetch the active storage profiles.
+For older score-server versions where the profile endpoint isn't available, the score-client will use the default (S3) storage profile implementation.
+This eliminates the need for users to ensure that score-client is running on a profile compatible with score-server.
+
can be configured with the `azure.blockSize` property.
\ No newline at end of file
diff --git a/score-client/pom.xml b/score-client/pom.xml
index 3afe0d24..547ca585 100644
--- a/score-client/pom.xml
+++ b/score-client/pom.xml
@@ -21,7 +21,7 @@ ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF S
bio.overturescore
- 5.9.0
+ 5.10.0../pom.xml
diff --git a/score-client/src/main/java/bio/overture/score/client/command/UploadCommand.java b/score-client/src/main/java/bio/overture/score/client/command/UploadCommand.java
index 926c2421..c1481355 100644
--- a/score-client/src/main/java/bio/overture/score/client/command/UploadCommand.java
+++ b/score-client/src/main/java/bio/overture/score/client/command/UploadCommand.java
@@ -23,14 +23,17 @@
import bio.overture.score.client.manifest.ManifestService;
import bio.overture.score.client.manifest.UploadManifest;
import bio.overture.score.client.upload.UploadService;
+import bio.overture.score.client.util.BeanUtil;
import com.beust.jcommander.Parameter;
import com.beust.jcommander.Parameters;
import lombok.extern.slf4j.Slf4j;
import lombok.val;
import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.context.ApplicationContext;
import org.springframework.context.annotation.Profile;
import org.springframework.stereotype.Component;
+import javax.annotation.PostConstruct;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.IOException;
@@ -64,9 +67,19 @@ public class UploadCommand extends RepositoryAccessCommand {
*/
@Autowired
private ManifestService manifestService;
- @Autowired
private UploadService uploader;
+ @Autowired
+ ApplicationContext appContext;
+
+ @Autowired
+ private BeanUtil beanUtil;
+
+ @PostConstruct
+ public void initializeStorageProfile() throws Exception{
+ uploader = (UploadService) beanUtil.getBeanForProfile(UploadService.class);
+ }
+
@Override
public int execute() throws Exception {
checkParameter(objectId != null || manifestResource != null, "One of --object-id or --manifest must be specified");
diff --git a/score-client/src/main/java/bio/overture/score/client/config/AzureConfig.java b/score-client/src/main/java/bio/overture/score/client/config/AzureConfig.java
index 4ee838df..bd3a09b3 100644
--- a/score-client/src/main/java/bio/overture/score/client/config/AzureConfig.java
+++ b/score-client/src/main/java/bio/overture/score/client/config/AzureConfig.java
@@ -19,24 +19,19 @@
import bio.overture.score.client.upload.UploadService;
import bio.overture.score.client.upload.azure.AzureUploadService;
+import bio.overture.score.client.upload.s3.S3UploadService;
import bio.overture.score.client.util.AzurePresignedUrlValidator;
import bio.overture.score.client.util.PresignedUrlValidator;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Profile;
+// Note: Name the beans as profile-name+class-type to be able to dynamically fetch beans at runtime based on score-server profiles
@Configuration
-@Profile("azure")
public class AzureConfig {
@Bean
- public UploadService uploadService() {
- return new AzureUploadService();
- }
-
+ public UploadService azUploadService(){ return new AzureUploadService();}
@Bean
- public PresignedUrlValidator urlValidator() {
- return new AzurePresignedUrlValidator();
- }
-
+ public AzurePresignedUrlValidator azPresignedUrlValidator(){return new AzurePresignedUrlValidator();}
}
diff --git a/score-client/src/main/java/bio/overture/score/client/config/ProfileConfig.java b/score-client/src/main/java/bio/overture/score/client/config/ProfileConfig.java
new file mode 100644
index 00000000..334e60da
--- /dev/null
+++ b/score-client/src/main/java/bio/overture/score/client/config/ProfileConfig.java
@@ -0,0 +1,72 @@
+package bio.overture.score.client.config;
+
+import bio.overture.score.client.exception.NotRetryableException;
+import bio.overture.score.core.model.StorageProfiles;
+import lombok.NonNull;
+import lombok.extern.slf4j.Slf4j;
+import lombok.val;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.beans.factory.annotation.Qualifier;
+import org.springframework.beans.factory.annotation.Value;
+import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
+import org.springframework.context.annotation.Bean;
+import org.springframework.context.annotation.Configuration;
+import org.springframework.http.HttpEntity;
+import org.springframework.http.HttpHeaders;
+import org.springframework.http.HttpMethod;
+import org.springframework.web.client.RestTemplate;
+
+@Configuration
+@Slf4j
+@ConditionalOnProperty(value="isTest", havingValue="false")
+public class ProfileConfig {
+
+ @Autowired
+ private RestTemplate serviceTemplate;
+
+ @Value("${storage.url}")
+ @NonNull
+ private String endpoint;
+
+ @Autowired
+ @Value("${defaultProfile:collaboratory}")
+ private String defaultProfile;
+
+ @Autowired
+ @Value("${isTest}")
+ private boolean isTest = false;
+
+ @Qualifier("clientVersion")
+ @Autowired
+ @NonNull
+ String clientVersion;
+
+ @Bean
+ public String storageProfile(){
+ String profile = getStorageProfile();
+ return profile;
+ }
+
+ private String getStorageProfile() {
+ log.debug("get profile endpoint: "+endpoint);
+ try{
+ String storageProfile = serviceTemplate.exchange(endpoint + "/profile", HttpMethod.GET, defaultEntity(), String.class).getBody();
+ return storageProfile;
+ }catch(NotRetryableException nre ){
+ log.error("received exception when getting profiles: " + nre.getMessage());
+ }
+ return StorageProfiles.getProfileValue(defaultProfile);
+ }
+
+
+ private HttpEntity
+
](http://slack.overture.bio)
+[
](https://github.com/overture-stack/score/blob/develop/LICENSE)
+[
](code_of_conduct.md)
-
+> 
+