{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# MDF Connect Client Tutorial" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "from mdf_connect_client import MDFConnectClient\n", "\n", "# These imports are only necessary for the examples in this notebook;\n", "# they are not necessary for regular use of the client.\n", "from datetime import datetime\n", "import time" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Table of contents\n", "\n", "- [Overview/instantiation](#MDF-Connect-Client)\n", "- [Mandatory inputs](#Mandatory-inputs)\n", "- [Recommended inputs](#Recommended-inputs)\n", "- [Optional inputs](#Optional-inputs)\n", "- [Advanced inputs](#Advanced-inputs)\n", "- [Submitting a dataset](#Submitting-a-dataset)\n", "- [Checking submission status](#Checking-submission-status)\n", "- [Curating a submission](#Curating-a-submission)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## MDF Connect Client\n", "The MDF Connect Client (`MDFConnectClient`) is a class designed to help you submit datasets to MDF Connect using Python. When you instantiate the Client, it will attempt to authenticate you with Globus automatically. You cannot use MDF Connect anonymously.\n", "\n", "Note: While you can access and modify the internal variables of a client (for example, `mdfcc.mdf`), it is recommended that you instead only use the helper functions. This tutorial accesses those variables only for display purposes.\n", "\n", "**IMPORTANT**: To submit data to MDF Connect, you must have an account recognized by Globus Auth (including Google, ORCiD, many academic institutions, or a [free Globus ID](https://www.globusid.org/create)). Additionally, you must be in the [MDF Connect](https://www.globus.org/app/groups/cc192dca-3751-11e8-90c1-0a7c735d220a/about) Globus Group." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Note:** You will see the comment `# NBVAL_SKIP` in some of these examples. It is an administrative flag. It does not affect the content of the example and can be safely ignored." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### MDFCC Constructor (`MDFConnectClient`)\n", "It is recommended that you use the helper functions (detailed below) to construct your MDF Connect submission. However, if you have already assembled all or part of your submission, you can pre-load your client with the appropriate metadata.\n", "\n", "Note: If you have the entire submission already prepared, you can skip to [Submitting a dataset](#Submitting-a-dataset) and pass your submission directly to `submit_dataset()`.\n", "\n", "#### Optional arguments:\n", "- `test` (boolean): When `True`, enables test mode. When `False`, disables test mode. For more information about test mode, see `set_test()`.\n", "\n", "#### Advanced optional arguments (developer use only):\n", "- `service_instance` (string): The instance of the MDF Connect service to use. Normal users should not alter this value.\n", "- `authorizer` (Authorizer): A valid, authenticated Authorizer from the Globus SDK. Normal users do not need to change this value. Accepted Authorizers are:\n", " - globus_sdk.RefreshTokenAuthorizer\n", " - globus_sdk.ClientCredentialsAuthorizer\n", " - globus_sdk.NullAuthorizer (This Authorizer will fail authentication.)\n", "\n", "The advanced arguments may cause issues, and are not recommended for normal users." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "mdfcc = MDFConnectClient(test=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `logout`\n", "If you want to log out and invalidate your current login, you can call `logout()`. This method clears your current submission and invalidates your current authentication tokens.\n", "\n", "Once this method is called, you must create a new MDF Connect Client in order to interact with MDF Connect." ] }, { "cell_type": "code", "execution_count": 56, "metadata": {}, "outputs": [], "source": [ "# mdfcc.logout()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Mandatory inputs\n", "\n", "- [`create_dc_block`](#create_dc_block)\n", "- [`add_data_source`](#add_data_source)\n", "\n", "These helpers are for inputs that are mandatory to provide. All required arguments for these required inputs must be supplied. Your submission will be rejected if you do not provide this information." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `create_dc_block`\n", "The `dc` (DataCite) block is mandatory for all submissions. `create_dc_block()` helps create the `dc` block for you.\n", "\n", "#### Required arguments:\n", "- `title` (string): The title of the dataset.\n", "- `authors` (string or list of strings): The authors of the dataset, in one of these forms:\n", " - \"Givenname Familyname\"\n", " - \"Familyname, Givenname\"\n", " - \"Familyname; Givenname\"\n", "\n", "#### Arguments with defaults:\n", "- `publisher` (string): The publisher of this dataset (*not* an associated paper). The default is `\"Materials Data Facility\"`.\n", "- `publication_year` (integer or string): The year the dataset was published. The default is the current year.\n", "- `resource_type` (string): The type of resource. Except in unusual cases, this should be `\"Dataset\"`. The default is `\"Dataset\"`. Unless you know that your submission needs a different value, please leave it as the default.\n", "\n", "#### Optional arguments (not present by default):\n", "- `affiliations` (string or list of strings): The affiliations of the authors, in the same order as the authors. If a different number of affiliations are given, all affiliations will be applied to all authors. Multiple author affiliations can be given as a list. (See examples below for more details.)\n", "- `description` (string): A description of the dataset.\n", "- `dataset_doi` (string): The DOI for this dataset (*not* an associated paper).\n", "- `related_dois` (string or list of strings): DOIs related to this dataset, such as an associated paper's DOI. This *does not* include a DOI for the dataset itself. \n", "- `subjects` (string or list of strings): Subjects (in Datacite terminology) or tags related to the dataset.\n", "\n", "\n", "If you understand the DataCite schema, you can also add other keyword arguments corresponding to DataCite fields. Additional information on DataCite fields is available from the [official DataCite website](https://schema.datacite.org/meta/kernel-4.1/).\n", "\n", "You cannot clear the `dc` block. You can overwrite the `dc` block by calling this method again." ] }, { "cell_type": "code", "execution_count": 57, "metadata": {}, "outputs": [], "source": [ "# Extra affiliations examples\n", "# Assume we have three authors: Alice, Bob, and Cathy\n", "authors = [\"Fromnist, Alice\", \"Fromnist; Bob\", \"Cathy Multiples\"]\n", "\n", "# If all authors are from NIST:\n", "affiliations = \"NIST\"\n", "# Equivalent to [\"NIST\", \"NIST\", \"NIST\"]\n", "\n", "# If all authors are from both NIST and UChicago:\n", "affiliations = [\"NIST\", \"UChicago\"]\n", "# Equivalent to [[\"NIST\", \"UChicago\"], [\"NIST\", \"UChicago\"], [\"NIST\", \"UChicago\"]]\n", "\n", "# If Alice and Bob are from NIST, Cathy is from NIST and UChicago:\n", "affliliations = [\"NIST\", \"NIST\", [\"NIST\", \"UChicago\"]]\n", "# This is the only way to express these affiliations\n", "\n", "# This is incorrect! If applying affiliations to all authors, lists must not be nested.\n", "# These apply to all authors because there are 4 affiliations for 3 authors.\n", "affiliations = [\"NIST\", [\"NIST\", \"UChicago\"], \"Argonne\", \"Oak Ridge\"]\n", "# Do not use this format, it is incorrect for this number of authors." ] }, { "cell_type": "code", "execution_count": 58, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "{'titles': [{'title': 'Sample Submission for Tutorial'}],\n", " 'creators': [{'creatorName': 'Smith, Foo',\n", " 'familyName': 'Smith',\n", " 'givenName': 'Foo',\n", " 'affiliations': ['The International Institute of Data']},\n", " {'creatorName': 'Smith, Bar',\n", " 'familyName': 'Smith',\n", " 'givenName': 'Bar',\n", " 'affiliations': ['The International Institute of Data']},\n", " {'creatorName': 'Baz, Smith;',\n", " 'familyName': 'Baz',\n", " 'givenName': 'Smith;',\n", " 'affiliations': ['The International Institute of Data']}],\n", " 'publisher': 'The Journal of Datasets',\n", " 'publicationYear': '2000',\n", " 'resourceType': {'resourceTypeGeneral': 'Dataset', 'resourceType': 'Dataset'},\n", " 'descriptions': [{'description': 'This is an example submission.',\n", " 'descriptionType': 'Other'}],\n", " 'identifier': {'identifier': '10.1234/dataset', 'identifierType': 'DOI'},\n", " 'relatedIdentifiers': [{'relatedIdentifier': '10.1234/paper1',\n", " 'relatedIdentifierType': 'DOI',\n", " 'relationType': 'IsPartOf'},\n", " {'relatedIdentifier': '10.1234/paper2',\n", " 'relatedIdentifierType': 'DOI',\n", " 'relationType': 'IsPartOf'}],\n", " 'subjects': [{'subject': 'Examples'}, {'subject': 'Datasets'}]}" ] }, "execution_count": 58, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdfcc.create_dc_block(title=\"Sample Submission for Tutorial\",\n", " authors=[\"Foo Smith\", \"Smith, Bar\", \"Smith; Baz\"],\n", " affiliations=[\"The International Institute of Data\"],\n", " publisher=\"The Journal of Datasets\",\n", " publication_year=2000,\n", " description=\"This is an example submission.\",\n", " dataset_doi=\"10.1234/dataset\",\n", " related_dois=[\"10.1234/paper1\", \"10.1234/paper2\"],\n", " subjects=[\"Examples\", \"Datasets\"])\n", "mdfcc.dc" ] }, { "cell_type": "code", "execution_count": 59, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'titles': [{'title': 'Tutorial Example Two'}],\n", " 'creators': [{'creatorName': 'Smith, Foo',\n", " 'familyName': 'Smith',\n", " 'givenName': 'Foo',\n", " 'affiliations': ['Foo University']},\n", " {'creatorName': 'Smith, Bar',\n", " 'familyName': 'Smith',\n", " 'givenName': 'Bar',\n", " 'affiliations': ['The Bureau of Bar']}],\n", " 'publisher': 'Materials Data Facility',\n", " 'publicationYear': '2020',\n", " 'resourceType': {'resourceTypeGeneral': 'Dataset', 'resourceType': 'Dataset'}}" ] }, "execution_count": 59, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdfcc.create_dc_block(title=\"Tutorial Example Two\",\n", " authors=[\"Foo Smith\", \"Smith, Bar\"],\n", " affiliations=[\"Foo University\", \"The Bureau of Bar\"])\n", "\n", "mdfcc.dc" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `add_data_source`\n", "Some kind of data is mandatory for all submissions. `add_data_source()` will add a data location to your dataset. This action is cumulative, so each calls adds more data. Subsequent calls do not overwrite.\n", "\n", "#### Required arguments:\n", "- `data_source` (string): The location of the data.\n", "\n", "You can add data located at a Globus endpoint, HTTP(S) link, or on Google Drive. MDF Connect will extract data located in archives, including zip files.\n", "\n", "- Globus endpoint: `globus://endpoint_id/path/to/data` or you can copy the \"Get link\" link from the Globus Web App. Your Globus account must have permission to read the data.\n", "- HTTP(S): Copy the link to the data file (NOT a landing page) exactly. The data must be accessible without authentication.\n", "- Google Drive: `googledrive:///path/from/shared/location`. You must share the data with materialsdatafacility@gmail.com.\n", "\n", "To clear all data from the submission, call `clear_data_sources()`." ] }, { "cell_type": "code", "execution_count": 60, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['https://dl.dropboxusercontent.com/u/12345/abcdef',\n", " 'googledrive:///mydata.zip',\n", " 'globus://1a2b3c/data/']" ] }, "execution_count": 60, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdfcc.add_data_source(\"https://dl.dropboxusercontent.com/u/12345/abcdef\")\n", "mdfcc.add_data_source([\"googledrive:///mydata.zip\", \"globus://1a2b3c/data/\"])\n", "mdfcc.data_sources" ] }, { "cell_type": "code", "execution_count": 61, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[]" ] }, "execution_count": 61, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdfcc.clear_data_sources()\n", "mdfcc.data_sources" ] }, { "cell_type": "code", "execution_count": 62, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['https://www.globus.org/app/transfer?origin_id=e38ee745-6d04-11e5-ba46-22000b92c6ec&origin_path=%2Fcitrine_mdf_demo%2Falloy.pbe%2FAlFe%2F']" ] }, "execution_count": 62, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# This is the actual test data, using a Globus Web App link\n", "mdfcc.add_data_source(\"https://www.globus.org/app/transfer?origin_id=e38ee745-6d04-11e5-ba46-22000b92c6ec&origin_path=%2Fcitrine_mdf_demo%2Falloy.pbe%2FAlFe%2F\")\n", "mdfcc.data_sources" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Recommended inputs\n", "\n", "- [`add_tag`](#add_tag)\n", "- [`add_index`](#add_index)\n", "- [`add_service`](#add_service)\n", "- [`set_test`](#set_test)\n", "- [`add_organization`](#add_organization)\n", "\n", "These helpers are for inputs that MDF recommends you provide or consider, but are not required." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `add_tag`\n", "`add_tag()` will add tags (also known as \"subjects\" in the DataCite schema, or \"keywords\") to your dataset.\n", "\n", "`add_tag(\"tag\")` is equivalent to `create_dc_block(..., subjects=[\"tag\"])`. This method exists for convenience of managing tags.\n", "\n", "#### Required arguments:\n", "- `tag` (string or list of strings): The tag to add.\n", "\n", "To clear all of the tags added with `add_tag()`, call `clear_tags()`. Note that `clear_tags()` does not remove tags set through `create_dc_block()`." ] }, { "cell_type": "code", "execution_count": 63, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['example']" ] }, "execution_count": 63, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdfcc.add_tag(\"example\")\n", "mdfcc.tags" ] }, { "cell_type": "code", "execution_count": 64, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[]" ] }, "execution_count": 64, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdfcc.clear_tags()\n", "mdfcc.tags" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `add_index`\n", "To extract JSON, CSV, YAML, XML, or Excel files, MDF Connect requires a mapping that translates the file into MDF schema format. `add_index()` will add a mapping for a specific data type to your submission.\n", "\n", "#### Required arguments:\n", "- `data_type` (string): The type of data being mapped. Supported types include:\n", " - `json`\n", " - `csv`\n", " - `yaml`\n", " - `xml`\n", " - `excel`\n", " - `filename` (This type is special; see below.)\n", "- `mapping` (dictionary of strings): The mapping of MDF fields to your data type's fields (see below).\n", "\n", "#### Arguments with defaults:\n", "- `delimiter` (string): For tabular data (ex. CSV), the column delimiter. The default is \",\" (comma).\n", "- `na_values` (string or list of strings): Values to treat as \"data missing\" entries. The default for tabular data (ex. CSV), v is blank and space, while the default for other types (ex. JSON) is nothing (no values will be discarded).\n", "\n", "#### About `mapping`:\n", "Mappings must be dictionaries, where the key is the MDF schema field (expressed in dot notation) and the value is the data's field or column name. \"Dot notation\" means one string that uses a period between dictionary levels. For example, `block.field.subfield` is the dot notation equivalent of `my_dict[\"block\"][\"field\"][\"subfield\"]` in Python.\n", "The exception to this is `filename` mapping. To extract data from a file's name, create a regular expression that returns the correct information. The mapping field is still the associated MDF field in dot notation, but the mapping value is the regular expression you created.\n", "\n", "Fields with missing data will be ignored. If you have multiple schemas for one data type in one dataset, you can combine the mappings safely.\n", "\n", "Each data type can only have one associated mapping, so multiple calls with the same data type will overwrite. Calls with different data types will not overwrite. To clear all the mappings, call `clear_index()`.\n", "\n", "For more information on the MDF schemas, see the [official schema repository](https://github.com/materials-data-facility/data-schemas)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For the following example, assume we're submitting a dataset that contains a JSON file structured like this:\n", "```json\n", "{\n", " \"my_data\": {\n", " \"mat\": {\n", " \"comp\": \"H\"\n", " },\n", " \"atom_num\": 1\n", " },\n", " \"space_grp\": 10\n", "}\n", "```" ] }, { "cell_type": "code", "execution_count": 65, "metadata": {}, "outputs": [], "source": [ "# This is the mapping we would use to get the JSON file into MDF format.\n", "mapping = {\n", " \"material.composition\": \"my_data.mat.comp\",\n", " \"crystal_structure.number_of_atoms\": \"my_data.atom_num\",\n", " \"crystal_structure.space_group_number\": \"space_grp\",\n", " # We could add another field here, if we had multiple JSON schemas.\n", " \"dft.converged\": \"dft_info.conv\"\n", " # This field would be ignored by MDF Connect in this submission because the field doesn't exist in the data.\n", "}" ] }, { "cell_type": "code", "execution_count": 66, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'json': {'mapping': {'material.composition': 'my_data.mat.comp',\n", " 'crystal_structure.number_of_atoms': 'my_data.atom_num',\n", " 'crystal_structure.space_group_number': 'space_grp',\n", " 'dft.converged': 'dft_info.conv'}}}" ] }, "execution_count": 66, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdfcc.add_index(\"json\", mapping)\n", "mdfcc.index" ] }, { "cell_type": "code", "execution_count": 67, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{}" ] }, "execution_count": 67, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdfcc.clear_index()\n", "mdfcc.index" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `add_service`\n", "MDF Connect has integrations to submit data to other community services, as well as additional MDF-related options. To automatically submit your dataset to an integrated service, use `add_service()`.\n", "\n", "#### Required arguments:\n", "- `service` (string): One service to push your dataset to. Integrated services include:\n", " - `mdf_publish`, the MDF publication service with DOI minting\n", " - `citrine`, industry-partnered machine-learning specialists\n", " - `mrr`, the NIST Materials Resource Registry\n", "\n", "#### Arguments with defaults:\n", "- `parameters` (dictionary): Optional, service-specific parameters. Fields include:\n", " - For `mdf_publish`:\n", " - publication_location (string): The Globus Endpoint and path on which to save the published files. The default will publish onto MDF resources.\n", " - For `citrine`:\n", " - public (boolean): When `True`, the data will be made public. Otherwise, the data will be inaccessible. The default is `True`.\n", "\n", "This action is cumulative, so subsequent calls will add more services, not overwrite previous.\n", "\n", "To clear all the service selections from your submission, call `clear_services()`." ] }, { "cell_type": "code", "execution_count": 68, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'citrine': True}" ] }, "execution_count": 68, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdfcc.add_service(\"citrine\")\n", "mdfcc.services" ] }, { "cell_type": "code", "execution_count": 69, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{}" ] }, "execution_count": 69, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdfcc.clear_services()\n", "mdfcc.services" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `set_test`\n", "You can use `set_test()` to create a test submission as a dry-run for MDF Connect. The submission will go through the normal processing, but the results will not be submitted to the normal locations. This flag is a great way to tell if your submission will process the way you want it to.\n", "\n", "#### Required arguments:\n", "- `test` (boolean): When `True`, enables test mode. When `False`, disables test mode.\n", "\n", "#### About test mode:\n", "Test datasets are submitted to test/sandbox/temporary resources instead of \"production\" resources, including the following. These setting override all other parameters.\n", "- Tests are ingested into the `mdf-test` search index\n", "- Tests are given a sandbox DOI, which is not permanent (if `mdf_publish` is a requested service)\n", "- Tests are not made public on Citrination (if `citrine` is a requested service)\n", "- Tests are given a special `source_id` by prepending `_test_`" ] }, { "cell_type": "code", "execution_count": 70, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "False" ] }, "execution_count": 70, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdfcc.set_test(False)\n", "mdfcc.test" ] }, { "cell_type": "code", "execution_count": 71, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "True" ] }, "execution_count": 71, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdfcc.set_test(True)\n", "mdfcc.test" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `add_organization`\n", "`add_organization()` marks your submission for an organization. This action is cumulative, so each call adds more organizations. Subsequent calls do not overwrite. Organizations may modify the parameters of your submission, such as mandating curation. More information about specific organizations can be found using [MDF Forge](https://mdf-forge.readthedocs.io/en/master/mdf_forge.html#mdf_forge.Forge.describe_organization).\n", "\n", "#### Required arguments:\n", "- `organization` (string or list of strings): The organization to add.\n", "\n", "Organizations automatically add their parent organizations. Organizations not registered with MDF will be discarded.\n", "\n", "To clear your organizations, call `clear_organizations()`." ] }, { "cell_type": "code", "execution_count": 72, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'organizations': ['CHiMaD']}" ] }, "execution_count": 72, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdfcc.add_organization(\"CHiMaD\")\n", "mdfcc.mdf" ] }, { "cell_type": "code", "execution_count": 73, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{}" ] }, "execution_count": 73, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdfcc.clear_organizations()\n", "mdfcc.mdf" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Optional inputs\n", "\n", "- [`set_custom_block`](#set_custom_block)\n", "- [`set_custom_descriptions`](#set_custom_descriptions)\n", "- [`set_base_acl`](#set_base_acl)\n", "- [`set_dataset_acl`](#set_dataset_acl)\n", "- [`set_source_name`](#set_source_name)\n", "- [`set_incremental_update`](#set_incremental_update)\n", "- [`add_data_destination`](#add_data_destination)\n", "- [`set_external_uri`](#set_external_uri)\n", "- [`create_mrr_block`](#create_mrr_block)\n", "\n", "These helpers are for inputs that are optional and can be skipped if you aren't interested in providing them." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `set_custom_block`\n", "The `custom` block is an area for you to add your own custom metadata, if it isn't covered by the MDF schema. It can be set by calling `set_custom_block()`.\n", "\n", "#### Required arguments:\n", "- `custom_fields` (dictionary): Custom field-value pairs for your dataset.\n", "\n", "You are allowed ten keys in your custom dictionary. You may additionally add descriptions of your fields by creating a new field called \"\\[field\\]\\_desc\" with the string description inside. You can also add descriptions by calling `set_custom_descriptions()`.\n", "\n", "Note that, unlike the `index` mappings, you supply the actual values for the dataset-level `custom` block.\n", "\n", "Subsequent calls will overwrite your `custom` block. You can clear the `custom` block by passing in an empty dictionary." ] }, { "cell_type": "code", "execution_count": 74, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'quench_method': 'water', 'quench_method_desc': 'The method of quenching'}" ] }, "execution_count": 74, "metadata": {}, "output_type": "execute_result" } ], "source": [ "custom_values = {\n", " \"quench_method\": \"water\",\n", " \"quench_method_desc\": \"The method of quenching\"\n", "}\n", "mdfcc.set_custom_block(custom_values)\n", "mdfcc.custom" ] }, { "cell_type": "code", "execution_count": 75, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{}" ] }, "execution_count": 75, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdfcc.set_custom_block({})\n", "mdfcc.custom" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `set_custom_descriptions`\n", "To add descriptions for your `custom` block fields, you can call `set_custom_descriptions()`.\n", "\n", "#### Required arguments:\n", "- `custom_descriptions` (dictionary): The custom fields and descriptions. The dictionary fields must be the same as your `custom` block fields, and the values must be their descriptions.\n", "\n", "Every field in `custom` can have a description, but descriptions are not allowed without a corresponding field.\n", "\n", "Subsequent calls will overwrite the descriptions you provide. To clear descriptions, you have to use `set_custom_block()`." ] }, { "cell_type": "code", "execution_count": 76, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'quench_method': 'water'}" ] }, "execution_count": 76, "metadata": {}, "output_type": "execute_result" } ], "source": [ "custom_values = {\n", " \"quench_method\": \"water\"\n", "}\n", "mdfcc.set_custom_block(custom_values)\n", "mdfcc.custom" ] }, { "cell_type": "code", "execution_count": 77, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'quench_method': 'water', 'quench_method_desc': 'The method of quenching'}" ] }, "execution_count": 77, "metadata": {}, "output_type": "execute_result" } ], "source": [ "custom_desc = {\n", " \"quench_method\": \"The method of quenching\"\n", "}\n", "mdfcc.set_custom_descriptions(custom_desc)\n", "mdfcc.custom" ] }, { "cell_type": "code", "execution_count": 78, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{}" ] }, "execution_count": 78, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdfcc.set_custom_block({})\n", "mdfcc.custom" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `set_base_acl`\n", "`set_base_acl()` sets the Access Control List for all the data in this submission. Anyone in this list can read the dataset entry, record entries, and files in the dataset.\n", "\n", "#### Required arguments:\n", "- acl (string or list of strings): The Access Control List. The ACL must contain either the Globus UUIDs of users and/or groups allowed to access the submission, or `\"public\"` to make the submission open to everyone. The default ACL is `\"public\"`.\n", "\n", "You can reset the ACL to the default with `clear_base_acl()`.\n", "\n", "#### *Warning*:\n", "The identities listed in the `base_acl` of your submission can always see your submission, including dataset entry, even if they are not listed in the `dataset_acl`. This means that **if you do not specify a `base_acl`**, because it defaults to `\"public\"`, **your entire dataset will be public.**\n", "\n", "MDF encourages you to make your data public, but if you do not want it public you must specify this value." ] }, { "cell_type": "code", "execution_count": 79, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'acl': ['UUID1', 'UUID2']}" ] }, "execution_count": 79, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdfcc.set_base_acl([\"UUID1\", \"UUID2\"])\n", "mdfcc.mdf" ] }, { "cell_type": "code", "execution_count": 80, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{}" ] }, "execution_count": 80, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdfcc.clear_base_acl()\n", "mdfcc.mdf" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `set_dataset_acl`\n", "`set_dataset_acl()` sets the Access Control List for just the dataset entry in MDF Search. Anyone in this list can see that dataset entry but _not_ the record entries or files (unless they are also in the `base_acl`, see above).\n", "\n", "#### Required arguments:\n", "- acl (string or list of strings): The Access Control List. The ACL must contain either the Globus UUIDs of users and/or groups allowed to access the dataset entry, or `\"public\"` to make the dataset entry open to everyone. By default, the dataset ACL is empty (which means that the base ACL is the only effective permission list).\n", "\n", "You can reset the ACL to the default with `clear_dataset_acl()`." ] }, { "cell_type": "code", "execution_count": 81, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['public']" ] }, "execution_count": 81, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdfcc.set_dataset_acl([\"public\"])\n", "mdfcc.dataset_acl" ] }, { "cell_type": "code", "execution_count": 82, "metadata": {}, "outputs": [], "source": [ "mdfcc.clear_dataset_acl()\n", "mdfcc.dataset_acl" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `set_source_name`\n", "`set_source_name()` sets the `source_name` of your dataset. By default, the `source_name` is generated based on the title of your dataset (as set in the `dc` block). If your title is long or otherwise unwieldy to type or remember, setting a custom `source_name` can help.\n", "\n", "#### Required arguments:\n", "- `source_name` (string): The desired `source_name`, which must be unique for new datasets.\n", "\n", "Please note that your source name will be cleaned when submitted to MDF Connect, so the actual source_name may differ from this value. Additionally, the `source_id` (which is the `source_name` plus version) is required to fetch the status of a submission. `check_status()` can handle this for you.\n", "\n", "You can reset the `source_name` to the default by calling `clear_source_name()`." ] }, { "cell_type": "code", "execution_count": 83, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'source_name': 'my_foobar_dataset'}" ] }, "execution_count": 83, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdfcc.set_source_name(\"my_foobar_dataset\")\n", "mdfcc.mdf" ] }, { "cell_type": "code", "execution_count": 84, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{}" ] }, "execution_count": 84, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdfcc.clear_source_name()\n", "mdfcc.mdf" ] }, { "cell_type": "code", "execution_count": 85, "metadata": {}, "outputs": [], "source": [ "# Here we're setting a unique source_name, so the submission will create a new dataset.\n", "mdfcc.set_source_name(\"tutorial_submission_{}\".format(int(time.time())))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `set_incremental_update`\n", "`set_incremental_update()` makes this submission an incremental update of a previous submission. Incremental updates use the same submission metadata, except for whatever you specify in the new submission.\n", "\n", "For example, if you submit an incremental update and only include a ``data_source``, the submission will run as if you copied the DC block and other metadata into the submission, but used the new ``data_source``. The new submission is processed normally, including data download and metadata extraction. (To update only the dataset metadata, use [`submit_dataset_metadata_update()`](#submit_dataset_metadata_update) when submitting.)\n", "\n", "**Note:**\n", "You must still set ``update=True`` when submitting an incremental update.\n", "\n", "#### Required arguments:\n", "- `source_id` (string): The ``source_id`` of the previous submission to update and resubmit.\n", "\n", "You can unmark an incremental update by calling this method with the `source_id` set to `False`." ] }, { "cell_type": "code", "execution_count": 86, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'old_submission_v4.2'" ] }, "execution_count": 86, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdfcc.set_incremental_update(\"old_submission_v4.2\")\n", "mdfcc.incremental_update" ] }, { "cell_type": "code", "execution_count": 87, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "False" ] }, "execution_count": 87, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdfcc.set_incremental_update(False)\n", "mdfcc.incremental_update" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `add_data_destination`\n", "`add_data_destination()` will add secondary storage locations for your dataset. MDF Connect will automatically use Globus Transfer to send your data to all of the data destinations you list. Destinations must be Globus Endpoints and the path must be writable by `mdf_dataset_submission`, which is the MDF Connect Globus account. (This account will show up as `c17f27bb-f200-486a-b785-2a25e82af505@clients.auth.globus.org`.)\n", "\n", "#### Required arguments:\n", "- `data_destination` (string or list of strings): The Globus Endpoint for backing up the dataset. The destinations must be formatted as `globus://endpoint_id/path/to/destination` or you can copy the \"Get link\" link from the Globus Web App.\n", "\n", "You can clear all data destinations by calling `clear_data_destinations()`." ] }, { "cell_type": "code", "execution_count": 88, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['globus://e38ee745-6d04-11e5-ba46-22000b92c6ec/my_data/mdf_submissions/']" ] }, "execution_count": 88, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdfcc.add_data_destination(\"globus://e38ee745-6d04-11e5-ba46-22000b92c6ec/my_data/mdf_submissions/\")\n", "mdfcc.data_destinations" ] }, { "cell_type": "code", "execution_count": 89, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[]" ] }, "execution_count": 89, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdfcc.clear_data_destinations()\n", "mdfcc.data_destinations" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `set_external_uri`\n", "If your dataset is already hosted at another data repository, you can use `set_external_uri()` to point at it. This link will be added to the dataset entry in MDF Search.\n", "\n", "#### Required arguments:\n", "- `uri` (string): The link to the external landing page for this dataset.\n", "\n", "You can clear the external URI by calling `clear_external_uri()`." ] }, { "cell_type": "code", "execution_count": 90, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'https://example.com'" ] }, "execution_count": 90, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdfcc.set_external_uri(\"https://example.com\")\n", "mdfcc.external_uri" ] }, { "cell_type": "code", "execution_count": 91, "metadata": {}, "outputs": [], "source": [ "mdfcc.clear_external_uri()\n", "mdfcc.external_uri" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `create_mrr_block`\n", "`create_mrr_block()` adds data for the NIST Materials Resource Registry into your submission.\n", "\n", "Currently, you must build a dictionary with the appropriate fields yourself in order to attach MRR metadata. Helpful arguments, in line with `create_dc_block()`, are intended to be added in the future.\n", "\n", "#### Required arguments:\n", "- `mrr_data` (dictionary): The Materials Resource Registry metadata.\n", "\n", "You can clear the `mrr` block by passing in an empty dictionary." ] }, { "cell_type": "code", "execution_count": 92, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'dataOrigin': 'experiment'}" ] }, "execution_count": 92, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdfcc.create_mrr_block({\"dataOrigin\": \"experiment\"})\n", "mdfcc.mrr" ] }, { "cell_type": "code", "execution_count": 93, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{}" ] }, "execution_count": 93, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdfcc.create_mrr_block({})\n", "mdfcc.mrr" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Advanced inputs\n", "\n", "- [`set_passthrough`](#set_passthrough)\n", "- [`set_project_block`](#set_project_block)\n", "- [`set_curation`](#set_curation)\n", "- [`set_conversion_config`](#set_conversion_config)\n", "\n", "These helpers are for advanced inputs that most users don't need to worry about." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `set_passthrough`\n", "`set_passthrough()` sets the pass-through (or no-extract) flag for your submission.\n", "\n", "Caution: The pass-through flag will cause metadata from your dataset's files to not be extracted by MDF Connect, so only high-level dataset metadata will be available in MDF Search. _This flag is only intended for datasets that cannot be extracted._\n", "\n", "HTTP(S) data sources are not supported when the pass-through flag is set.\n", "\n", "#### Required arguments:\n", "- `passthrough` (boolean): When `False`, the dataset will be processed normally. When `True`, the metadata in the dataset files will not be extracted." ] }, { "cell_type": "code", "execution_count": 94, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "True" ] }, "execution_count": 94, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdfcc.set_passthrough(True)\n", "mdfcc.no_extract" ] }, { "cell_type": "code", "execution_count": 95, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "False" ] }, "execution_count": 95, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdfcc.set_passthrough(False)\n", "mdfcc.no_extract" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `set_project_block`\n", "`set_project_block()` sets project-specific metadata on your dataset entry. The project block is a special area for specific metadata, that must be registered with the MDF. If you have a project block defined in the MDF schema, you can set that metadata for your dataset entry in this way.\n", "\n", "#### Required arguments:\n", "- `project` (string): The name of the project block in MDF.\n", "- `data` (dictionary): The metadata for the project block.\n", "\n", "You can clear this block by passing in an empty `data` argument." ] }, { "cell_type": "code", "execution_count": 96, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'example_project': {'field': 'value'}}" ] }, "execution_count": 96, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdfcc.set_project_block(\"example_project\", {\"field\": \"value\"})\n", "mdfcc.projects" ] }, { "cell_type": "code", "execution_count": 97, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{}" ] }, "execution_count": 97, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdfcc.set_project_block(\"example_project\", None)\n", "mdfcc.projects" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `set_curation`\n", "To trigger curation of your submission, use `set_curation()`. An approved curator (see below) must accept your submission before it will be indexed in MDF Search (and published with MDF Publish or sent to any other services, if applicable). Normally, this flag is set by an organization's rules, and not by an end-user, but you can set it yourself if you like.\n", "\n", "##### About approved curators:\n", "If your organization has set the curation flag, the approved curators are the managers and admins of the organization's permission groups. If you manually set the curation flag, the approved curators are based on your Access Control List (see [`set_acl()`](#set_acl)); anyone you list directly in your ACL can curate, as well as managers and admins of any group you list.\n", "If you set your ACL to \"public\" then anyone can curate your submission.\n", "\n", "#### Required arguments:\n", "- `curation` (boolean): When `False`, the dataset will be fully processed automatically and not require approval. When `True`, the dataset will go through metadata extraction and then require an approved curator to accept it before ingesting to any service (including MDF Search and MDF Publish).\n", "\n", "Remember that your organization (set with [`add_organization()`](#add_organization)) may force curation. Setting the curation flag to `False` does not override your organization's rules." ] }, { "cell_type": "code", "execution_count": 98, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "True" ] }, "execution_count": 98, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdfcc.set_curation(True)\n", "mdfcc.curation" ] }, { "cell_type": "code", "execution_count": 99, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "False" ] }, "execution_count": 99, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdfcc.set_curation(False)\n", "mdfcc.curation" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `set_extraction_config`\n", "`set_extraction_config()` sets advanced configuration parameters for your submission in the `extraction_config` block. These options are intended for advanced users and/or special-case datasets. Most submissions do not need to worry about these parameters.\n", "\n", "#### Required arguments:\n", "- `config` (dictionary): The extraction configuration options.\n", "\n", "You can clear this block by passing in an empty dictionary." ] }, { "cell_type": "code", "execution_count": 100, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'group_by_dir': True}" ] }, "execution_count": 100, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdfcc.set_extraction_config({\"group_by_dir\": True})\n", "mdfcc.extraction_config" ] }, { "cell_type": "code", "execution_count": 101, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{}" ] }, "execution_count": 101, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdfcc.set_extraction_config({})\n", "mdfcc.extraction_config" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Submitting a dataset\n", "- [`get_submission`](#get_submission)\n", "- [`reset_submission`](#reset_submission)\n", "- [`submit_dataset`](#submit_dataset)\n", "- [`submit_dataset_metadata_update`](#submit_dataset_metadata_update)\n", "\n", "After you have created your submission with the above helpers, you can submit and check your submission with these helpers." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `get_submission`\n", "`get_submission()` shows you your current submission, as it will be sent to MDF Connect. This method is a great way to check for any errors.\n", "\n", "#### Return value:\n", "- A dictionary containing the current submission." ] }, { "cell_type": "code", "execution_count": 102, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'dc': {'titles': [{'title': 'Tutorial Example Two'}],\n", " 'creators': [{'creatorName': 'Smith, Foo',\n", " 'familyName': 'Smith',\n", " 'givenName': 'Foo',\n", " 'affiliations': ['Foo University']},\n", " {'creatorName': 'Smith, Bar',\n", " 'familyName': 'Smith',\n", " 'givenName': 'Bar',\n", " 'affiliations': ['The Bureau of Bar']}],\n", " 'publisher': 'Materials Data Facility',\n", " 'publicationYear': '2020',\n", " 'resourceType': {'resourceTypeGeneral': 'Dataset',\n", " 'resourceType': 'Dataset'}},\n", " 'data_sources': ['https://www.globus.org/app/transfer?origin_id=e38ee745-6d04-11e5-ba46-22000b92c6ec&origin_path=%2Fcitrine_mdf_demo%2Falloy.pbe%2FAlFe%2F'],\n", " 'test': True,\n", " 'update': False,\n", " 'mdf': {'source_name': 'tutorial_submission_1580155419'}}" ] }, "execution_count": 102, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mdfcc.get_submission()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `reset_submission`\n", "If you need to clear away your entire submission, call `reset_submission()`. This is irreversible.\n", "\n", "Caution: This method will clear the current `source_id`, which means that you will have to keep track of any previous `source_id`s from other submissions to see their statuses." ] }, { "cell_type": "code", "execution_count": 103, "metadata": {}, "outputs": [], "source": [ "# mdfcc.reset_submission()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `submit_dataset`\n", "`submit_dataset()` will send your dataset to MDF Connect for indexing. You will get back the `source_id` if the submission is successful. The `source_id` is the unique identifier for your specific submission, and can be used to check the status of your submission later. The `source_id` is also saved to the client.\n", "\n", "#### Optional arguments:\n", "- `update` (boolean): If you wish to submit this dataset after submitting it previously, set this to `True`. If this is the first submission, leave this `False`. The default is `False`.\n", "- `submission` (dictionary): If you have assembled your own MDF Connect submission without this client, you can submit it by passing the dictionary in here. By default, the submission made in the client will be used.\n", "- `reset` (boolean): If True, the submission will be cleared after the submission attempt, with `reset_submission()`. The test flag will be preserved. The default is `False`. Caution: This flag will clear your `source_id`, which means that you will have to keep track of it manually to check your submission's status.\n", "\n", "#### Return value:\n", "- A dictionary with the following submission information:\n", " - `success` (boolean): `True` if the submission was successfully sent to MDF Connect. `False` otherwise.\n", " - `source_id` (string): The `source_id` of the submission, from MDF Connect. This value may be `None` or an old ID if the submission failed.\n", " - `error` (string): If the submission failed, the reason for failure. If the submission suceeded, this will be `None` instead." ] }, { "cell_type": "code", "execution_count": 104, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'source_id': '_test_tutorial_submission_1580155419_v1.1',\n", " 'success': True,\n", " 'error': None,\n", " 'status_code': 202}" ] }, "execution_count": 104, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# NBVAL_SKIP\n", "\n", "mdfcc.submit_dataset()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `submit_dataset_metadata_update`\n", "To submit only updates to an existing submission's dataset entry, use `submit_dataset_metadata_update()`. This includes updates to the DC block, such as the author list. You can create the metadata using the same helpers, but any helpers that set non-dataset metadata (such as [`add_data_source()`](#add_data_source)) will be ignored.\n", "\n", "To be clear, this method submits an update to a dataset entry only, and NOT the data or record entries. The submission you are updating must have completed processing successfully at the time you submit the update - you are not allowed to update a submission that is still processing, or failed processing.\n", "\n", "#### Required arguments:\n", "- `source_id` (string): The `source_id` of the dataset you wish to update. You must be the owner of the dataset.\n", "\n", "#### Optional arguments:\n", "- `metadata_update` (dictionary): If you have assembled the dataset metadata yourself, you can submit it here. This argument supersedes any data set through other methods. The default is `None`, to use the method-assembled data.\n", "- `reset` (boolean): If `True`, will clear the old metadata from the client. The `test` flag will be preserved. If `False`, the metadata will be preserved. The default is `False`." ] }, { "cell_type": "code", "execution_count": 109, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'success': True, 'error': None, 'status_code': 200}" ] }, "execution_count": 109, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# NBVAL_SKIP\n", "\n", "update_source_id = \"author_updatable_example_submission_v1.1\"\n", "new_metadata = {\n", " \"dc\": {\n", " \"titles\": [{\n", " \"title\": \"Updated Title\"\n", " }]\n", " }\n", "}\n", "mdfcc.submit_dataset_metadata_update(update_source_id, metadata_update=new_metadata)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Checking submission status\n", "- [`check_status`](#check_status)\n", "- [`check_all_submissions`](#check_all_submissions)\n", "\n", "After submitting a dataset to MDF Connect, you can see the status of the submission's processing with these helpers." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `check_status`\n", "To see the progress your submission is making, use `check_status()`. If you haven't cleared the submission from the client, you can use it without arguments to check the most recent submission status.\n", "\n", "#### Optional arguments:\n", "- `source_id` (string): The `source_id` of the submission you want to check on. If you don't supply a `source_id`, the ID of the last submission you made with the client will be used instead (an error will result if you have not submitted a dataset with the client yet and also don't supply an ID).\n", "- `short` (boolean): When `False`, a status summary will be printed for the submission. When `True`, an abbreviated summary containing only the minimum information will be printed (this is useful for checking many submissions at once). The default is `False`.\n", "- `raw` (boolean): When `False`, a nicely-formatted status summary will be printed to standard output. When `True`, the full status result will be returned instead (the full result is not recommended for direct human consumption). The default is `False`.\n", "\n", "#### Return value:\n", "- A dictionary containing the full submission status (only when `raw` is `True`)." ] }, { "cell_type": "code", "execution_count": 106, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Status of TEST submission _test_tutorial_submission_1580155419_v1.1 (Tutorial Example Two)\n", "Submitted by Jonathon Gaff at 2020-01-27T20:03:52.207133Z\n", "\n", "Submission initialization was successful.\n", "Connect data download was successful: 21 files will be grouped and extracted (from 0 archives).\n", "Data transfer to primary destination is in progress.\n", "Metadata extraction has not started yet.\n", "Dataset curation has not started yet.\n", "MDF Search ingestion has not started yet.\n", "Data transfer to secondary destinations has not started yet.\n", "MDF Publish publication has not started yet.\n", "Citrine upload has not started yet.\n", "Materials Resource Registration has not started yet.\n", "Post-processing cleanup has not started yet.\n", "\n", "This submission is still processing.\n", "\n" ] } ], "source": [ "# NBVAL_SKIP\n", "\n", "mdfcc.check_status()" ] }, { "cell_type": "code", "execution_count": 107, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Status of TEST submission _test_name_status_checking_example_v1.1 (Status Checking Example)\n", "Submitted by Jonathon Gaff at 2019-08-08T20:26:36.263611Z\n", "\n", "Submission initialization was successful.\n", "Connect data download was successful: 12 files will be converted (0 archives extracted).\n", "Data transfer to primary destination was successful.\n", "Metadata extraction was successful: 4 records parsed out of 4 groups.\n", "Dataset curation was not requested or required.\n", "MDF Search ingestion was successful.\n", "Data transfer to secondary destinations was not requested or required.\n", "MDF Publish publication was not requested or required.\n", "Citrine upload was not requested or required.\n", "Materials Resource Registration was not requested or required.\n", "Post-processing cleanup was successful.\n", "\n", "This submission is no longer processing.\n", "\n" ] } ], "source": [ "mdfcc.check_status(\"_test_name_status_checking_example_v1.1\")" ] }, { "cell_type": "code", "execution_count": 108, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "_test_name_status_checking_example_v1.1: This submission is no longer processing.\n" ] } ], "source": [ "mdfcc.check_status(\"_test_name_status_checking_example_v1.1\", short=True)" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "{'status': {'active': False,\n", " 'source_id': '_test_name_status_checking_example_v1.1',\n", " 'status_code': 'SMSMNSNNNNS',\n", " 'status_list': [{'signal': 'success',\n", " 'text': 'Submission initialization was successful.'},\n", " {'signal': 'success',\n", " 'text': 'Connect data download was successful: 12 files will be converted (0 archives extracted).'},\n", " {'signal': 'success',\n", " 'text': 'Data transfer to primary destination was successful.'},\n", " {'signal': 'success',\n", " 'text': 'Metadata extraction was successful: 4 records parsed out of 4 groups.'},\n", " {'signal': 'idle',\n", " 'text': 'Dataset curation was not requested or required.'},\n", " {'signal': 'success', 'text': 'MDF Search ingestion was successful.'},\n", " {'signal': 'idle',\n", " 'text': 'Data transfer to secondary destinations was not requested or required.'},\n", " {'signal': 'idle',\n", " 'text': 'MDF Publish publication was not requested or required.'},\n", " {'signal': 'idle', 'text': 'Citrine upload was not requested or required.'},\n", " {'signal': 'idle',\n", " 'text': 'Materials Resource Registration was not requested or required.'},\n", " {'signal': 'success', 'text': 'Post-processing cleanup was successful.'}],\n", " 'status_message': 'Status of TEST submission _test_name_status_checking_example_v1.1 (Status Checking Example)\\nSubmitted by Jonathon Gaff at 2019-08-08T20:26:36.263611Z\\n\\nSubmission initialization was successful.\\nConnect data download was successful: 12 files will be converted (0 archives extracted).\\nData transfer to primary destination was successful.\\nMetadata extraction was successful: 4 records parsed out of 4 groups.\\nDataset curation was not requested or required.\\nMDF Search ingestion was successful.\\nData transfer to secondary destinations was not requested or required.\\nMDF Publish publication was not requested or required.\\nCitrine upload was not requested or required.\\nMaterials Resource Registration was not requested or required.\\nPost-processing cleanup was successful.\\n',\n", " 'submission_time': '2019-08-08T20:26:36.263611Z',\n", " 'submitter': 'Jonathon Gaff',\n", " 'test': True,\n", " 'title': 'Status Checking Example'},\n", " 'success': True,\n", " 'status_code': 200}" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# This is not recommended for human consumption\n", "mdfcc.check_status(\"_test_name_status_checking_example_v1.1\", raw=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `check_all_submissions`\n", "If you want to see the status of all submissions you've made to MDF Connect, use `check_all_submissions()`. This method is helpful if you forget a submission's `source_id`, or you have multiple submissions processing at once.\n", "\n", "#### Optional arguments:\n", "- `verbose` (boolean): When `False`, a basic summary of your submissions will be printed. When `True`, the full status summary of each submission will be printed, in the same form as `check_status()`. The default is `False`. (This argument has no effect if `raw` is `True`.)\n", "- `active_only` (boolean): When `True`, only active submissions will be printed. The default is `False`.\n", "- `include_tests` (boolean): When `False`, only non-test submissions will be printed. The default is `True`.\n", "- `newer_than_date` (datetime or tuple of integers): Excludes submissions made before this date. Accepts a `datetime` object or a tuple of `(year, month, day)` integers. The default is `None`, to set no maximum age.\n", "- `older_than_date`: (datetime or tuple of ints): Excludes submissions made after this date. Accepts a `datetime` object or a tuple of `(year, month, day)` integers. The default is `None`, to set no minimum age.\n", "- `raw` (boolean): When `False`, the summary selected by `verbose` will be printed. When `True`, the full status result will be returned instead (the full result is not recommended for direct human consumption). The default is `False`.\n", "\n", "#### Return value:\n", "- A dictionary containing the full submission statuses (only when `raw` is `True`).\n", "\n", "#### Note about date filtering:\n", "Days are compared in UTC, at exactly 0:00 (12:00am). This means that the two dates cannot be the same, as they would filter out all submissions not made at exactly 0:00:00 on the chosen date. To see submissions made on a specific date, set the older_than filter one day away from the date in question. For example, to see submissions made on Feb 11, 2020, use `newer_than_date=(2020, 2, 11), older_than_date=(2020, 2, 12)`." ] }, { "cell_type": "code", "execution_count": 52, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "_test_tutorial_submission_1579021617_v1.1: Processing - Not started\n", "_test_abrehabiruk_virtual_db_v1.1: Processing - Processing\n", "_test_smaller_example_dataset_submission_v3-2: Processing - Processing\n", "_test_smaller_example_dataset_submission_v3-1: Processing - Processing\n", "_test_name_curation_task_example_v1.4: Processing - Processing\n", "_test_name_curation_rejecting_example_v1.4: Processing - Processing\n", "_test_name_curation_accepting_example_v4.2: Processing - Processing\n" ] } ], "source": [ "mdfcc.check_all_submissions(active_only=True)" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "_test_gaff_email_testing_v1.2: Not processing - Succeeded\n", "_test_gaff_email_testing_v1.1: Not processing - Cancelled\n", "_test_gaff_gdrive_verification_v2.1: Not processing - Failed\n" ] } ], "source": [ "# Filtering for submissions made between Jan 1, 2020 and Feb 10, 2020\n", "mdfcc.check_all_submissions(newer_than_date=datetime(2020, 5, 30), older_than_date=(2020, 7, 10))" ] }, { "cell_type": "code", "execution_count": 53, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "\n", "Status of TEST submission _test_tutorial_submission_1579021617_v1.1 (Tutorial Example Two)\n", "Submitted by Jonathon Gaff at 2020-01-14T17:07:00.063623Z\n", "\n", "Submission initialization has not started yet.\n", "Connect data download has not started yet.\n", "Data transfer to primary destination has not started yet.\n", "Metadata extraction has not started yet.\n", "Dataset curation has not started yet.\n", "MDF Search ingestion has not started yet.\n", "Data transfer to secondary destinations has not started yet.\n", "MDF Publish publication has not started yet.\n", "Citrine upload has not started yet.\n", "Materials Resource Registration has not started yet.\n", "Post-processing cleanup has not started yet.\n", "This submission is still processing.\n", "\n", "\n", "Status of TEST submission _test_abrehabiruk_virtual_db_v1.1 (Virtual Excited State Reference for the Discovery of Electronic Materials (VERDE Materials DB))\n", "Submitted by Jonathon Gaff at 2019-09-12T18:51:41.927599Z\n", "\n", "Submission initialization was successful.\n", "Connect data download was successful: 8417 files will be converted (1 archives extracted).\n", "Data transfer to primary destination was successful.\n", "Metadata extraction is in progress.\n", "Dataset curation has not started yet.\n", "MDF Search ingestion has not started yet.\n", "Data transfer to secondary destinations has not started yet.\n", "MDF Publish publication has not started yet.\n", "Citrine upload has not started yet.\n", "Materials Resource Registration has not started yet.\n", "Post-processing cleanup has not started yet.\n", "This submission is still processing.\n", "\n", "\n", "Status of TEST submission _test_smaller_example_dataset_submission_v3-2 (Smaller Example Dataset Submission)\n", "Submitted by Jonathon Gaff at 2018-12-05T20:56:45.927284Z\n", "\n", "Submission initialization is in progress.\n", "Connect data download has not started yet.\n", "Data transfer to primary destination has not started yet.\n", "Metadata extraction has not started yet.\n", "Dataset curation has not started yet.\n", "MDF Search ingestion has not started yet.\n", "Data transfer to secondary destinations has not started yet.\n", "MDF Publish publication has not started yet.\n", "Citrine upload has not started yet.\n", "Materials Resource Registration has not started yet.\n", "Post-processing cleanup has not started yet.\n", "This submission is still processing.\n", "\n", "\n", "Status of TEST submission _test_smaller_example_dataset_submission_v3-1 (Smaller Example Dataset Submission)\n", "Submitted by Jonathon Gaff at 2018-12-05T20:55:14.066883Z\n", "\n", "Submission initialization was successful.\n", "Connect data download was successful: 21 files will be processed (0 archives extracted).\n", "Data transfer to primary destination was successful: 9 records parsed out of 7 groups.\n", "Metadata extraction was successful.\n", "Dataset curation is in progress.\n", "MDF Search ingestion has not started yet.\n", "Data transfer to secondary destinations has not started yet.\n", "MDF Publish publication has not started yet.\n", "Citrine upload has not started yet.\n", "Materials Resource Registration has not started yet.\n", "Post-processing cleanup has not started yet.\n", "This submission is still processing.\n", "\n", "\n", "Status of TEST submission _test_name_curation_task_example_v1.4 (Curation Task Example)\n", "Submitted by Jonathon Gaff at 2020-01-14T16:48:27.414911Z\n", "\n", "Submission initialization was successful.\n", "Connect data download was successful: 12 files will be grouped and extracted (from 0 archives).\n", "Data transfer to primary destination was successful.\n", "Metadata extraction was successful: 4 metadata records extracted out of 4 file groups.\n", "Dataset curation is in progress.\n", "MDF Search ingestion has not started yet.\n", "Data transfer to secondary destinations has not started yet.\n", "MDF Publish publication has not started yet.\n", "Citrine upload has not started yet.\n", "Materials Resource Registration has not started yet.\n", "Post-processing cleanup has not started yet.\n", "This submission is still processing.\n", "\n", "\n", "Status of TEST submission _test_name_curation_rejecting_example_v1.4 (Curation Rejecting Example)\n", "Submitted by Jonathon Gaff at 2020-01-14T16:47:56.995810Z\n", "\n", "Submission initialization was successful.\n", "Connect data download was successful: 12 files will be grouped and extracted (from 0 archives).\n", "Data transfer to primary destination was successful.\n", "Metadata extraction was successful: 4 metadata records extracted out of 4 file groups.\n", "Dataset curation is in progress.\n", "MDF Search ingestion has not started yet.\n", "Data transfer to secondary destinations has not started yet.\n", "MDF Publish publication has not started yet.\n", "Citrine upload has not started yet.\n", "Materials Resource Registration has not started yet.\n", "Post-processing cleanup has not started yet.\n", "This submission is still processing.\n", "\n", "\n", "Status of TEST submission _test_name_curation_accepting_example_v4.2 (Curation Accepting Example)\n", "Submitted by Jonathon Gaff at 2020-01-14T16:48:14.775396Z\n", "\n", "Submission initialization was successful.\n", "Connect data download was successful: 12 files will be grouped and extracted (from 0 archives).\n", "Data transfer to primary destination was successful.\n", "Metadata extraction was successful: 4 metadata records extracted out of 4 file groups.\n", "Dataset curation is in progress.\n", "MDF Search ingestion has not started yet.\n", "Data transfer to secondary destinations has not started yet.\n", "MDF Publish publication has not started yet.\n", "Citrine upload has not started yet.\n", "Materials Resource Registration has not started yet.\n", "Post-processing cleanup has not started yet.\n", "This submission is still processing.\n" ] } ], "source": [ "mdfcc.check_all_submissions(verbose=True, include_tests=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Curating a submission\n", "- [`get_curation_task`](#get_curation_task)\n", "- [`get_available_curation_tasks`](#get_available_curation_tasks)\n", "- [`accept_curation_submission`](#accept_curation_submission)\n", "- [`reject_curation_submission`](#reject_curation_submission)\n", "\n", "When a submission has the curation flag set through [`set_curation()`](#set_curation) or through an organization's rules (see [`add_organization()`](#add_organization)), the dataset goes through metadata extraction but is temporarily stopped before ingestion to any other service (including MDF Search and MDF Publish, when applicable). An approved curator must review and accept the submission before it can proceed (see below). These helpers allow an approved curator to view and approve or reject submissions waiting for curation.\n", "\n", "##### About approved curators:\n", "If the dataset submitter's organization has set the curation flag, the approved curators are the managers and admins of the organization's permission groups. If the curation flag was manually set, the approved curators are based on the Access Control List (see [`set_acl()`](#set_acl)); anyone listed directly in the ACL can curate, as well as managers and admins of any group listed.\n", "If the ACL is \"public\" then anyone can curate the submission, but it will not show up in `get_available_curation_tasks()`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `get_curation_task`\n", "To see the details of a specific dataset that needs curation, use `get_curation_task()`. You must have permission to curate any submission you wish to view.\n", "\n", "#### Required arguments:\n", "- `source_id` (string): The `source_id` of the submission you want to view. If you don't know the `source_id`, you can use [`get_available_curation_tasks`](#get_available_curation_tasks) or ask the dataset submitter.\n", "\n", "#### Optional arguments:\n", "- `summary` (boolean): When `False`, the entire curation task, including the dataset entry and sample records, will be printed. When `True`, only a summary of the curation task will be printed. The default is `False`.\n", "- `raw` (boolean): When `False`, the curation task information selected by `summary` will be printed. When `True`, a dictionary containing the full curation task will be returned, regardless of `summary`. The default is `False`, which is recommended for direct human consumption.\n", "\n", "#### Return value:\n", "- A dictionary containing the full curation task (only when `raw` is `True`)." ] }, { "cell_type": "code", "execution_count": 54, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\n", " \"allowed_curators\": [\n", " \"public\"\n", " ],\n", " \"curation_start_date\": \"2020-01-14 16:51:04.459251\",\n", " \"dataset\": {\n", " \"data\": {\n", " \"endpoint_path\": \"globus://e38ee745-6d04-11e5-ba46-22000b92c6ec/MDF/mdf_connect/prod/data/_test_name_curation_task_example_v1.4/\",\n", " \"link\": \"https://app.globus.org/file-manager?origin_id=e38ee745-6d04-11e5-ba46-22000b92c6ec&origin_path=/MDF/mdf_connect/prod/data/_test_name_curation_task_example_v1.4/\"\n", " },\n", " \"dc\": {\n", " \"creators\": [\n", " {\n", " \"creatorName\": \"Name\",\n", " \"familyName\": \"Name\",\n", " \"givenName\": \"\"\n", " }\n", " ],\n", " \"publicationYear\": \"2020\",\n", " \"publisher\": \"Materials Data Facility\",\n", " \"resourceType\": {\n", " \"resourceType\": \"Dataset\",\n", " \"resourceTypeGeneral\": \"Dataset\"\n", " },\n", " \"titles\": [\n", " {\n", " \"title\": \"Curation Task Example\"\n", " }\n", " ]\n", " },\n", " \"mdf\": {\n", " \"acl\": [\n", " \"public\"\n", " ],\n", " \"ingest_date\": \"2020-01-14T16:50:55.265890Z\",\n", " \"resource_type\": \"dataset\",\n", " \"scroll_id\": 0,\n", " \"source_id\": \"_test_name_curation_task_example_v1.4\",\n", " \"source_name\": \"_test_name_curation_task_example\",\n", " \"version\": 1\n", " },\n", " \"services\": {}\n", " },\n", " \"extraction_summary\": \"4 records were extracted out of 4 groups from 12 files\",\n", " \"sample_records\": [\n", " {\n", " \"crystal_structure\": {\n", " \"number_of_atoms\": 4.0,\n", " \"space_group_number\": 221,\n", " \"stoichiometry\": \"AB3\",\n", " \"volume\": 66.87455107336443\n", " },\n", " \"dft\": {\n", " \"converged\": true,\n", " \"cutoff_energy\": 249.8,\n", " \"exchange_correlation_functional\": \"PAW_PBE\"\n", " },\n", " \"files\": [\n", " {\n", " \"data_type\": \"ASCII text\",\n", " \"filename\": \"OUTCAR\",\n", " \"globus\": \"globus://e38ee745-6d04-11e5-ba46-22000b92c6ec/MDF/mdf_connect/prod/data/_test_name_curation_task_example_v1.4/Ag3Al.cF4/OUTCAR\",\n", " \"length\": 98631,\n", " \"mime_type\": \"text/plain\",\n", " \"sha512\": \"0926ad16c90f477345254945938003f3e946c0d05c766802a28f57b71aba66bd9746987ea6d0c98aeeb454b0b6a28fd38ae9a5c4ce10f2d3172f40ec0af9964c\",\n", " \"url\": \"https://e38ee745-6d04-11e5-ba46-22000b92c6ec.e.globus.org/MDF/mdf_connect/prod/data/_test_name_curation_task_example_v1.4/Ag3Al.cF4/OUTCAR\"\n", " },\n", " {\n", " \"data_type\": \"ASCII text\",\n", " \"filename\": \"INCAR\",\n", " \"globus\": \"globus://e38ee745-6d04-11e5-ba46-22000b92c6ec/MDF/mdf_connect/prod/data/_test_name_curation_task_example_v1.4/Ag3Al.cF4/INCAR\",\n", " \"length\": 476,\n", " \"mime_type\": \"text/plain\",\n", " \"sha512\": \"b9d2bb98b4df98fd9b8583fa8135db7a4ef28d984b7ab4cf07ed11901564d8eed748568d0faa00ffe9674581a8c932012afe75cddc320c5657ae7503403ef255\",\n", " \"url\": \"https://e38ee745-6d04-11e5-ba46-22000b92c6ec.e.globus.org/MDF/mdf_connect/prod/data/_test_name_curation_task_example_v1.4/Ag3Al.cF4/INCAR\"\n", " },\n", " {\n", " \"data_type\": \"ASCII text\",\n", " \"filename\": \"POSCAR\",\n", " \"globus\": \"globus://e38ee745-6d04-11e5-ba46-22000b92c6ec/MDF/mdf_connect/prod/data/_test_name_curation_task_example_v1.4/Ag3Al.cF4/POSCAR\",\n", " \"length\": 743,\n", " \"mime_type\": \"text/plain\",\n", " \"sha512\": \"2118d83f6f1c29216d92fd2f0791da9d17d845fd49ac60df8648333296d8f6ec7b5f01b215c289459d5a2d63f4636549b96fb76b32c293d087dd9782bd90f92c\",\n", " \"url\": \"https://e38ee745-6d04-11e5-ba46-22000b92c6ec.e.globus.org/MDF/mdf_connect/prod/data/_test_name_curation_task_example_v1.4/Ag3Al.cF4/POSCAR\"\n", " }\n", " ],\n", " \"material\": {\n", " \"composition\": \"Al1Ag3\",\n", " \"elements\": [\n", " \"Ag\",\n", " \"Al\"\n", " ]\n", " },\n", " \"mdf\": {\n", " \"acl\": [\n", " \"public\"\n", " ],\n", " \"ingest_date\": \"2020-01-14T16:50:55.265890Z\",\n", " \"resource_type\": \"record\",\n", " \"scroll_id\": 1,\n", " \"source_id\": \"_test_name_curation_task_example_v1.4\",\n", " \"source_name\": \"_test_name_curation_task_example\",\n", " \"version\": 1\n", " }\n", " },\n", " {\n", " \"crystal_structure\": {\n", " \"number_of_atoms\": 4.0,\n", " \"space_group_number\": 123,\n", " \"stoichiometry\": \"AB3\",\n", " \"volume\": 67.20758519915091\n", " },\n", " \"dft\": {\n", " \"converged\": true,\n", " \"cutoff_energy\": 249.8,\n", " \"exchange_correlation_functional\": \"PAW_PBE\"\n", " },\n", " \"files\": [\n", " {\n", " \"data_type\": \"ASCII text\",\n", " \"filename\": \"OUTCAR\",\n", " \"globus\": \"globus://e38ee745-6d04-11e5-ba46-22000b92c6ec/MDF/mdf_connect/prod/data/_test_name_curation_task_example_v1.4/Ag3Al.cI2/OUTCAR\",\n", " \"length\": 631529,\n", " \"mime_type\": \"text/plain\",\n", " \"sha512\": \"04b4236e7bbdcc0688fa213ee65154a291c885f8fd11e77758d24066a3b0908c600b6ba97388e8ddef75f61f008e7d55150cf05ef99126392fc3602a04a2d384\",\n", " \"url\": \"https://e38ee745-6d04-11e5-ba46-22000b92c6ec.e.globus.org/MDF/mdf_connect/prod/data/_test_name_curation_task_example_v1.4/Ag3Al.cI2/OUTCAR\"\n", " },\n", " {\n", " \"data_type\": \"ASCII text\",\n", " \"filename\": \"INCAR\",\n", " \"globus\": \"globus://e38ee745-6d04-11e5-ba46-22000b92c6ec/MDF/mdf_connect/prod/data/_test_name_curation_task_example_v1.4/Ag3Al.cI2/INCAR\",\n", " \"length\": 476,\n", " \"mime_type\": \"text/plain\",\n", " \"sha512\": \"b9d2bb98b4df98fd9b8583fa8135db7a4ef28d984b7ab4cf07ed11901564d8eed748568d0faa00ffe9674581a8c932012afe75cddc320c5657ae7503403ef255\",\n", " \"url\": \"https://e38ee745-6d04-11e5-ba46-22000b92c6ec.e.globus.org/MDF/mdf_connect/prod/data/_test_name_curation_task_example_v1.4/Ag3Al.cI2/INCAR\"\n", " },\n", " {\n", " \"data_type\": \"ASCII text\",\n", " \"filename\": \"POSCAR\",\n", " \"globus\": \"globus://e38ee745-6d04-11e5-ba46-22000b92c6ec/MDF/mdf_connect/prod/data/_test_name_curation_task_example_v1.4/Ag3Al.cI2/POSCAR\",\n", " \"length\": 743,\n", " \"mime_type\": \"text/plain\",\n", " \"sha512\": \"d059850e59099a86e7c7d10dad5d1aeb673f093c1568291843487d9366fa2a3644775f94fd82a0a165f6d9bfcf7709806f3e663b17d3d2b50c175e88f535df7f\",\n", " \"url\": \"https://e38ee745-6d04-11e5-ba46-22000b92c6ec.e.globus.org/MDF/mdf_connect/prod/data/_test_name_curation_task_example_v1.4/Ag3Al.cI2/POSCAR\"\n", " }\n", " ],\n", " \"material\": {\n", " \"composition\": \"Al1Ag3\",\n", " \"elements\": [\n", " \"Ag\",\n", " \"Al\"\n", " ]\n", " },\n", " \"mdf\": {\n", " \"acl\": [\n", " \"public\"\n", " ],\n", " \"ingest_date\": \"2020-01-14T16:50:55.265890Z\",\n", " \"resource_type\": \"record\",\n", " \"scroll_id\": 2,\n", " \"source_id\": \"_test_name_curation_task_example_v1.4\",\n", " \"source_name\": \"_test_name_curation_task_example\",\n", " \"version\": 1\n", " }\n", " },\n", " {\n", " \"crystal_structure\": {\n", " \"number_of_atoms\": 20.0,\n", " \"space_group_number\": 146,\n", " \"stoichiometry\": \"AB3\",\n", " \"volume\": 336.3169877518198\n", " },\n", " \"files\": [\n", " {\n", " \"data_type\": \"ASCII text\",\n", " \"filename\": \"OUTCAR\",\n", " \"globus\": \"globus://e38ee745-6d04-11e5-ba46-22000b92c6ec/MDF/mdf_connect/prod/data/_test_name_curation_task_example_v1.4/Ag3Al.cP20/OUTCAR\",\n", " \"length\": 1702675,\n", " \"mime_type\": \"text/plain\",\n", " \"sha512\": \"b1c0f55b8175941d01c093a2e68434399fa2fbb8c83b94adbbf8d768fec9618f0e76b13f44ccba94266a769560c5125ca4ae8168e67b50fb2b52daaed74bb9a5\",\n", " \"url\": \"https://e38ee745-6d04-11e5-ba46-22000b92c6ec.e.globus.org/MDF/mdf_connect/prod/data/_test_name_curation_task_example_v1.4/Ag3Al.cP20/OUTCAR\"\n", " },\n", " {\n", " \"data_type\": \"ASCII text\",\n", " \"filename\": \"INCAR\",\n", " \"globus\": \"globus://e38ee745-6d04-11e5-ba46-22000b92c6ec/MDF/mdf_connect/prod/data/_test_name_curation_task_example_v1.4/Ag3Al.cP20/INCAR\",\n", " \"length\": 477,\n", " \"mime_type\": \"text/plain\",\n", " \"sha512\": \"ff55df884a9674c960596d9b4fb4e9c99f1f8e24f095029b85feec6026b5072047feeb307db8d8c4692656e954d666d66c0737517eae14e4bbf64510960c367f\",\n", " \"url\": \"https://e38ee745-6d04-11e5-ba46-22000b92c6ec.e.globus.org/MDF/mdf_connect/prod/data/_test_name_curation_task_example_v1.4/Ag3Al.cP20/INCAR\"\n", " },\n", " {\n", " \"data_type\": \"ASCII text\",\n", " \"filename\": \"POSCAR\",\n", " \"globus\": \"globus://e38ee745-6d04-11e5-ba46-22000b92c6ec/MDF/mdf_connect/prod/data/_test_name_curation_task_example_v1.4/Ag3Al.cP20/POSCAR\",\n", " \"length\": 2503,\n", " \"mime_type\": \"text/plain\",\n", " \"sha512\": \"1d27ae2226dde1302db27072e64d9c113048839fadb5ea0ca3ab0030c8cf891e0958ca9b67ce0a2afa63c03b7f2d4f5eb80afb9c921f8d30de8d331e183eabf4\",\n", " \"url\": \"https://e38ee745-6d04-11e5-ba46-22000b92c6ec.e.globus.org/MDF/mdf_connect/prod/data/_test_name_curation_task_example_v1.4/Ag3Al.cP20/POSCAR\"\n", " }\n", " ],\n", " \"material\": {\n", " \"composition\": \"Al5Ag15\",\n", " \"elements\": [\n", " \"Ag\",\n", " \"Al\"\n", " ]\n", " },\n", " \"mdf\": {\n", " \"acl\": [\n", " \"public\"\n", " ],\n", " \"ingest_date\": \"2020-01-14T16:50:55.265890Z\",\n", " \"resource_type\": \"record\",\n", " \"scroll_id\": 3,\n", " \"source_id\": \"_test_name_curation_task_example_v1.4\",\n", " \"source_name\": \"_test_name_curation_task_example\",\n", " \"version\": 1\n", " }\n", " }\n", " ],\n", " \"source_id\": \"_test_name_curation_task_example_v1.4\",\n", " \"submission_info\": {\n", " \"acl\": [\n", " \"public\"\n", " ],\n", " \"canon_destination\": \"globus://e38ee745-6d04-11e5-ba46-22000b92c6ec/MDF/mdf_connect/prod/data/_test_name_curation_task_example_v1.4/\",\n", " \"curation\": true,\n", " \"data_destinations\": [],\n", " \"data_sources\": [\n", " \"https://app.globus.org/file-manager?origin_id=e38ee745-6d04-11e5-ba46-22000b92c6ec&origin_path=%2Fcitrine_mdf_demo%2Falloy.pbe%2FAgAl%2F\"\n", " ],\n", " \"dataset_acl\": [\n", " \"public\"\n", " ],\n", " \"extraction_config\": {},\n", " \"index\": {\n", " \"file\": {\n", " \"globus_host\": \"globus://e38ee745-6d04-11e5-ba46-22000b92c6ec/MDF/mdf_connect/prod/data/_test_name_curation_task_example_v1.4/\",\n", " \"http_host\": \"https://e38ee745-6d04-11e5-ba46-22000b92c6ec.e.globus.org\",\n", " \"local_path\": \"/home/ubuntu/data/_test_name_curation_task_example_v1.4/\"\n", " }\n", " },\n", " \"no_extract\": false,\n", " \"services\": {\n", " \"mdf_search\": {\n", " \"index\": \"mdf-test\"\n", " }\n", " },\n", " \"submitter\": \"Jonathon Gaff\",\n", " \"test\": true,\n", " \"update\": true\n", " }\n", "}\n" ] } ], "source": [ "mdfcc.get_curation_task(\"_test_name_curation_task_example_v1.4\")" ] }, { "cell_type": "code", "execution_count": 55, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "_test_name_curation_task_example_v1.4 by Jonathon Gaff\n", "Waiting since 2020-01-14 16:51:04.459251\n", "4 records were extracted out of 4 groups from 12 files\n", "\n" ] } ], "source": [ "mdfcc.get_curation_task(\"_test_name_curation_task_example_v1.4\", summary=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `get_available_curation_tasks`\n", "To see all of the submissions you have permission to curate (excluding \"public\" submissions, which anyone can curate), use `get_available_curation_tasks()`.\n", "\n", "#### Optional arguments:\n", "- `summary` (boolean): When `False`, the entiretly of each curation task available to you, including the dataset entry and sample records, will be printed. When `True`, only a summary of each curation task will be printed. The default is `True`. The summaries are very useful to get an overview of tasks, but it is recommended to then use [`get_curation_task`](#get_curation_task) to view the details.\n", "- `raw` (boolean): When `False`, the curation task information selected by `summary` will be printed. When `True`, a dictionary containing the full curation tasks will be returned, regardless of `summary`. The default is `False`, which is recommended for direct human consumption.\n", "\n", "#### Return value:\n", "- A dictionary containing the full curation tasks (only when `raw` is `True`).\n", "\n", "**Note:** This helper does not show curation tasks that are \"public\", which anyone can curate, to minimize irrelevant results." ] }, { "cell_type": "code", "execution_count": 56, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "You have no open curation tasks.\n" ] } ], "source": [ "mdfcc.get_available_curation_tasks()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `accept_curation_submission`\n", "After reviewing a curation task, you can accept the submission with `accept_curation_submission()`. You must have permission to curate any submission you wish to accept.\n", "\n", "It is strongly recommended that you view submissions with [`get_curation_task`](#get_curation_task) before accepting or rejecting them.\n", "\n", "After a submission is accepted, it will continue processing and ingest to MDF Search (and any other applicable services, such as MDF Publish).\n", "\n", "#### Required arguments:\n", "- `source_id` (string): The `source_id` of the submission you wish to accept.\n", "\n", "#### Optional arguments:\n", "- `reason` (string): The reason for accepting this submission. If a reason is not provided, a generic acceptance message will be used instead.\n", "- `prompt` (boolean): When `True`, you will be prompted to confirm acceptance of the submission with a task summary. When `False`, confirmation will not be required. The default is `True`, and it is recommended to review this summary to avoid errors.\n", "- `raw` (boolean): When `False`, the completion result will be printed. When `True`, a dictionary of the completion result will be returned. The default is `False`.\n", "\n", "#### Return value:\n", "- A dictionary containing the completion result (only when `raw` is `True`)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Note:** This sample curation task was submitted outside of this notebook, to make a simple example of accepting a submission. It cannot be re-accepted." ] }, { "cell_type": "code", "execution_count": 57, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Are you sure you want to accept the following submission?\n", "_test_name_curation_accepting_example_v4.2 by Jonathon Gaff\n", "Waiting since 2020-01-14 16:51:05.598132\n", "4 records were extracted out of 4 groups from 12 files\n", "\n", "\n", "Confirm accepting submission [yes/no]: yes\n", "\n", "What is the reason for accepting this submission?\n", "\tThis dataset shows substantive results.\n", "\n", "Submission accepted with reason: This dataset shows substantive results.\n" ] } ], "source": [ "# NBVAL_SKIP\n", "\n", "mdfcc.accept_curation_submission(\"_test_name_curation_accepting_example_v4.2\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `reject_curation_submission`\n", "After reviewing a curation task, you can reject the submission with `reject_curation_submission()`. You must have permission to curate any submission you wish to reject.\n", "\n", "It is strongly recommended that you view submissions with [`get_curation_task`](#get_curation_task) before accepting or rejecting them.\n", "\n", "After a submission is rejected, it will fail and permanently stop processing. It will not be ingested to MDF Search or any other services (such as MDF Publish).\n", "\n", "#### Required arguments:\n", "- `source_id` (string): The `source_id` of the submission you wish to reject.\n", "\n", "#### Optional arguments:\n", "- `reason` (string): The reason for rejecting this submission. If a reason is not provided, a generic rejection message will be used instead. It is recommended to provide this argument to help the submitter understand why you rejected the submission.\n", "- `prompt` (boolean): When `True`, you will be prompted to confirm rejection of the submission with a task summary. When `False`, confirmation will not be required. The default is `True`, and it is recommended to review this summary to avoid errors.\n", "- `raw` (boolean): When `False`, the completion result will be printed. When `True`, a dictionary of the completion result will be returned. The default is `False`.\n", "\n", "#### Return value:\n", "- A dictionary containing the completion result (only when `raw` is `True`)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Note:** This sample curation task was submitted outside of this notebook, to make a simple example of rejecting a submission. It cannot be re-rejected." ] }, { "cell_type": "code", "execution_count": 58, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Are you sure you want to reject the following submission?\n", "_test_name_curation_rejecting_example_v1.4 by Jonathon Gaff\n", "Waiting since 2020-01-14 16:50:25.421009\n", "4 records were extracted out of 4 groups from 12 files\n", "\n", "\n", "Confirm rejecting submission [yes/no]: yes\n", "\n", "What is the reason for rejecting this submission?\n", "\tThis submission misused the analysis technique.\n", "\n", "Submission rejected with reason: This submission misused the analysis technique.\n" ] } ], "source": [ "# NBVAL_SKIP\n", "\n", "mdfcc.reject_curation_submission(\"_test_name_curation_rejecting_example_v1.4\")" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.1" } }, "nbformat": 4, "nbformat_minor": 2 }