Documentation

Older and stable releases are also available here: https://cdstar.gwdg.de/release/

Building CDSTAR requires a Java JDK (Java 11 or newer) and Maven. The CDSTAR source distribution ships with a Maven wrapper script (./mvnw or ./mvnw.bat) that fetches the correct version of Maven and sould be preferred over whatever Maven version is offered as a system package by your distribution.

CDStar can be compiled into a cdstar.war file and run within a servlet container, but this is not recommended and not officially supported. CDStar also does not offer any built-in daemonizing capabilities. If you want to run cdstar as a long-running background process, use proper system tools like systemd, supervisord or traditional init.d scripts and start-stop-daemon as a last resort.

To begin, we import some helpful functions from the 'requests' module, define our API base URL and create our first archive.

CDStar returns JSON most of the time, so we can use requests.Response.json() to parse the response directly into a python dictionary. In this case, we are only interested in the id field of the response. This string identifies our archive within a vault and can be used to build the archive URL. Alternatively, we could just follow the Location header.

The archive is still empty. We can list its content with simple a GET request.

As you can see, there are no files in this archive. Let’s change that and upload some files.

There are mutiple ways to populate an archive. The simplest way is to send multipart/form-data POST requests to the archive URL. Each file upload with a name that start with a slash (e.g. /example.txt) creates a new file in our archive.

The response is JSON again and contains a list of all files that changed during the last request. We can use this info to double-check if everything was uploaded correctly.

Now we want to attach some meta attributes to our archive and the file we just uploaded. We send just another POST request to the same URL, but this time we use form-fields starting with meta: to define new meta attribute on the archive or a file within the archive.

Just like the file upload example from above, we get back a report of everything that changed.

Let us have a look at our archive again and also request file and meta-attribute listings this time.

The file and meta fields are hidden by default and only included if you add with=files,meta as a query parameter. For large archives, you can even filter and paginate the returned information. See getArchiveInfo for details.

Each file within an archive has its own URL, for example /myVault/ab587f42c2570a884/some/file.txt. You can create, read, update or delete individual files by sending the respective PUT, GET, POST or DELETE requests to these URLs, which is sometimes a lot easier than working with the form-based API described earlier, especially from within scripts or programmable REST clients.

First, let’s upload a new file to the archive. Just PUT the raw file content to the file URL.

If you need more control over whether a file should be overwritten or not, you can add one of the following conditional headers to your request:

You should check for 412 Precondition Failed errors in your application if you use these headers.

Once the file is stored in the archive, you can retrieve it using the same URL.

This downloads the entire file and stores it locally. You can also request parts of the file (using Range headers) and make your request conditional (If-Match, If-None-Match, If-Modified-Since, If-Unodified-Since and If-Range headers are fully supported).

Instead of the actual file content, you can also request the file attributes or meta-attributes via the info and meta sub-resources.

And finally: Deleting individual files is just a plain and simple DELETE request.

Thats it for now. To be continued …​

CDStar API Endpoints make use of the following standard HTTP request methods:

Some proxies restrict or lack support for certain HTTP methods, such as DELETE. In this case, a client may send a POST request with a non-standard X-HTTP-Method-Override header instead. The value of this header is used as a server-side override for the actual HTTP method.

Each of the API Endpoints defines a number of possible HTTP response status codes and their meaning. The following list summarizes all status codes used by this API and provides a general description.

API Endpoints may accept request parameters of various types, either via the query string part of the request URL, or as fields within a multipart/form+data formatted POST request, or both. In any case, each parameter is associated with a value type and interpreted according to the following table:

Glob patterns are a simple way to filter or match file-names within an archive against a specific pattern. There is no real standard for glob patterns and existing implementations differ slightly. This is why CDStar implements its own subset of the most commonly used rules:

If the whole pattern starts with the path separator / (forward slash), then the entire path is matched against the pattern. Otherwise, a partial match at the end of the path is sufficient. The pattern *.pdf for example would return all PDF files within an archive, but /*.pdf would only return PDF files located directly within the root folder.

As mentioned above, single wildcards only match within a path segment, which means both ? and * do not expand across path separators (/). The pattern docs/*.pdf would find /docs/file.pdf but not /docs/subfolder/file.pdf. Use two adjacent asterisks (e.g docs/**.pdf) to include subfolders in your search.

HTTP Basic Authentication is a stateless and simple authentication scheme most suitable for scripting or simple client applications. Username and password are transmitted with each request in cleartext, so this scheme should NOT be used over unencrypted connections.

Some realms may require a fully qualified username in the form of username@realm, but most realms also accept unqualified logins. If the username itself contains an @, then it MUST be qualified to avoid ambiguity.

Token authentication is handled via an Authorization: bearer <token> header. Alternatively, the non-standard X-Token header or token query parameter can be used, but these are not recommended. Acquiring a token is not part of this API and depends heavily on the configured token realm (e.g. JWTRealm). For this example we assume that the client already obtained an access token.

In order to embed resources into HTML pages (e.g. images) or provide time-limited download links, a special token with limited access rights can be attached to the URL of GET requests via the token query parameter. As with access tokens, the method to obtain such tokens is not part of this API.

Subjects are encoded as strings and matched against the current user context using following subject qualifier syntax:

If multiple realms are configured, then group and user names should be qualified with a realm name to avoid naming conflicts between realms. Unqualified names are still allowed, but they will match against any realm with a matching user or group.

Fully qualified names have the form name@realm. For example, alice from the ldap realm would be alice@ldap. Only the last occurrence of the @ character is recognized, so identifiers with @ in them (e.g. email addresses) are allowed. In fact, if the local part of a subject identifier contains an @, then the subject MUST be qualified with a realm to avoid ambiguity.

Permissions regarding a specific vault. If assigned globally, they have the from vault:{vaultName}:{permissionName}.

Archives are protected by an access control list (ACL) which grands permissions to specific subjects (see Subjects and Realms). If assigned globally, they have the form archive:{vaultName}:{archiveId}:{permissionName}.

Archive permissions are very fine-grained and most actions require more than one permission. For example, a user with only read_file permission on an archive would not be able to read any files, because the load permission is also required to load the archive in the first place. To simplify access control for common use-cases, permission sets were introduced. Each set bundles a number of permissions that are usually granted together, and can be assigned just like normal permissions.

Permission sets have upper-case names to distinguish them from normal permissions. The following matrix shows all pre-defined permission sets and their corresponding permissions.

The MANAGE set is intended for management and reporting jobs. These are usually only interested in the meta-data of an archive, not the content. The set therefore inherits LIST instead of READ or even WRITE to protect user data by default. While clients with this permission set would be able to grant more permissions to themselves, these changes would show up in audit logs and be accountable.

Vaults are usually configured to grant OWNER permissions to the $owner subject for new archives automatically. This allows the archive creator to work with the newly created archive and perform most actions, with the notable exception of changing the owner. Giving archives away is usually a task reserved for higher privilege accounts. This permission set is not limited or otherwise tied to the $owner subject, though. It can be given to other subjects, or revoked from the owner. Revoking permissions from the owner is a common pattern to make archives read-only after publishing.

Get basic information about the cdstar instance as well as a list of all vaults accessible by the current user.

Return health and performance metrics about the service.

List vaults accessible by the current user. This is the same as Service Info.

Get information about a vault.

Perform a search over all archives and files within a vault using the configured search backend. Only results that are visible to the currently logged in user are returned.

List IDs of archives stored in this vault.

Up to limit IDs are returned per request. IDs are ordered in a stable but otherwise implemention specifc way (usually lexographical). If the scroll parameter is a non-empty string, then only IDs ordered after the given string are returned. This can be used to scroll through all IDs of a vault in an efficient manner.

By default, this API will return all IDs that were ever created in this vault, including IDs of archives that were removed or are not load-able by the current user. This mode requires list vault permission or the vault to be public.

In strict mode, archive manifests are actually loaded from storage and only IDs of archives that are load-able by the current user are returned. This mode is less efficient, but does not require list permissions on the vault. Use with caution.

This API is NOT transactional and may reflect changes made by other clients as soon as they happen.

Create a new archive, owned by the current user.

If the request body contains form data, the new archive is immediately populated according to Update Archive.

Get information about an archive or list its content.

When accessing a snapshot, information that is not part of the snapshot (e.g. owner or ACL) will be read from the current archive state.

Export (a subset of) all files in an archive as a single download.

The file format is specified by the export parameter. Currently only zip is implemented. More formats are planned (e.g. BagIt, tar, tar.gz and more).

Update an existing archive or snapshot.

Request form data is interpreted as a list of commands and applied in order. The order is significant. For example, a file can be uploaded, then copied, then annotated with metadata, all in the same request. Make sure your HTTP client preserves the order of form fields when generating the request body.

Commands that contain a {filename} placeholder operate on files within the archive. The filename must start with a slash (/) in order to be recognized. If the filename also ends with a slash, it usually affects all files with that prefix. Be careful.

File uploads only work with multipart/form-data and are not recommended for large files. Prefer Upload file for anything larger than a couple of MB. Uploading a large number of small files may be faster using this api, though. Your mileage may vary.

Snapshots are read-only, but setting a new profile is supported.

Remove an existing archive and all snapshots. This requires delete permissions on the archive.

List files within an archive or snapshot. This endpoint supports the same parameters as Get Archive Info to filter or paginate the list of files.

Download a single file from an archive or snapshot.

This endpoint supports ranged requests and conditional headers such as If-(None-)Match, If-(Un)modified-Since, If-Range and Range, as well as HEAD requests. The ETag value is calculated from the files digest hash, if known.

Highly accessed files in publicy readable archives may be served from a different location (e.g. S3 or CDN). Clients should follow redirects (e.g. 307 Temporary Redirect) according to the HTTP standard.

During explixit transactions, and while a file upload is currently in progress, GET requests will fail with an "IncompleteWrite" error. HEAD requests are allowed, though. The Content-Length header will report the current upload size.

Get FileInfo for a single file. For multiple files, [fileList] is usually faster.

Directly upload a new file to an archive, or overwrite an existing file.

If a Content-Type header is missing or equals application/x-autodetect, then the media type is guessed from the filename extention.

The conditional headers If-Match: * or If-None-Match: * can be used to force update-only or create-only behavior.

Upload errors can only be detected properly if either Content-Length header is set, or Transfer-Encoding: chunked is used. If less than the expected number of bytes are transmitted, the file is considered incomplete and the transaction will fail.

During explicit transactions (see Transaction Management), failed uploads will leave the file in an incomplete state. The upload must be repeated or resumed before committing. See Resume file upload for details. Conflicting operations, for example reading the file content or fetching its info, will fail until the file was completely updated or removed. HEAD requests to the files URL are allowed, though.

Resume a failed or aborted file upload.

After a failed Upload file request during an explicit transactions (see Transaction Management), the client may choose to resume the upload instead of uploading the entire file again or removing it.

To do so, send a PATCH request with Content-Type: application/vnd.cdstar.resume and a Range header with a single byte range, either bytes=startByte- or bytes=startByte-endByte (see RFC-2616). The startByte index must match the current remote file size, as returned by a HEAD request to the Download file API. The endByte index is optional, but recommended as an additional saveguard. It should match the target file size.

A file is considered complete once the PUT or PATCH request completes without errors. Within a single transaction, failing uploads can be resumed repeatedly until all data is transmitted or the transaction runs into a timeout.

Do not use this api to upload files in small chunks. A successfull PUT or PATCH request will compute digests, which is an expensive operation. Always try to upload the entire file in one go, if possible.

Remove a single file from an archive. This requires change_files permissions on the archive.

Return metadata attributes for an archive or snapshot. The same information can also received as part of a Get Archive Info request by using the with=meta switch.

Replace the metadata of an archive with a new MetaAttributes document. To clear all attributes, just send an empty document (e.g. {}).

Return metadata attributes for a single file within an archive or snapshot. The same information can also received as part of a Get file info request by using the with=meta switch.

Replace the metadata of a file within an archive with a new MetaAttributes document. To clear all attributes, just send an empty document (e.g. {}).

Return the local access control list of this archive as an AclInfo document. The same information can also be received as part of a Get Archive Info request by using the with=acl switch.

Replace all entries of the local access control list with entries from this AclInfo document.

Create a new Archive from a ZIP or TAR file.

For compressed TAR files, make sure to provide a suitable Content-Encoding header. Supported algorithms include gz, bzip2, xz, and deflate.

Note that importing compressed ZIP or TAR archives requires a significant amount of work on server-side after the upload completed, which may cause some clients to time-out before a response can be sent. Make sure to increase the read time-outs for your client before uploading large archives.

Import files from a zip or tar file into an existing archive. See Import from ZIP/TAR for details.

Create a new snapshot.

Delete a snapshot. This requires delete permissions on the archive and is irreversable. The name of a deleted snapshot cannot be used to create a new snapshot.

Get a list of snapshots that exist for this archive, ordered by creation date, then name.

Start a new transaction. See Transaction Management for details.

Request information about a running transaction.

Commit a running transaction. All changes made with this transaction ID are persisted and new transactions will be able to see the changes. The commit may fail, in wich case not changes will be persisted at all. Partial commits never happen.

Renew a running transaction. This resets the transaction timeout and ensures that the transaction is not rolled back automatically for the next TransactionInfo.ttl seconds.

Close a running transation by rolling it back. All changes made with this transaction ID are discarded.

The realm is configured directly in the cdstar main configuration. Here is an example showing most options:

Unqualified groups and user-names are qualified with the configured default domain of the realm (e.g. alice is turned into alice@static). Fully qualified names (e.g. alice@otherRealm) are also accepted, even if the domain does not match the current realm.

If no password is defined for a user, then the user will not be able to authenticate against this realm. Permissions, roles and groups still apply.

A secure password-hash can be generated with the java -cp cdstar.jar de.gwdg.cdstar.auth.realm.StaticRealm tool.

Warning: cache.expire is enforced by the cache implementation, which might allows entries to survive longer than expected on Java 8 if the cache is mostly idle. If prompt expiration is important and the expiration time is very short, make sure to run on Java 9 or newer.

The JWTRealm class can be configured as a realm or regular plugin and allows users to authenticate via signed JWTs.

This plugin supports multiple JWT issuers with different settings at the same time. Tokens are matched against configured issuers based in their iss claim. Tokens without an iss claim or with no matching issuer configuration will be matched against the default issuer, if defined.

Each issuer MUST define at least one of hmac, rsa, ecdsa or jwks to be able to verify signed tokens. Unsigned tokens are not supported and will be rejected.

Because JWT is a very loose standard and the available claims may differ a lot between token providers, this plugin allows to verify tokens and extract information dynamically using SpEL expressions. Token claims are available as a claims map which maps claim names to com.auth0.jwt.interfaces.Claim instances, or via the hasClaim(name), getBool(name, default), getLong(name, default), getDouble(name, default), getString(name, default), getStringList(name), getClaim(name, type, default) and getClaimList(name, innerType) helper methods. These methods will return null or an empty list on any errors (missing claim, wrong type) and automatically convert between single and list claims. If a single value is requested for a list claim, the first value is returned.

If the issuer is configured with trusted: true, then the following rules and expressions are automatically configured for a realm:

Events are send to consumers synchronously and in the order they appear, which means that there is at most one HTTP connection per consumer at any given time. The service behind the configured URL should expect requests like the following:

A consumer may respond with 200 OK, 202 Accepted or 204 No Content to signal success. The response body should be empty and other headers (including cookies) are ignored.

Redirects with 30x response codes are followed according to normal HTTP client rules, but discouraged.

Consumers that are busy or unresponsive can answer with 503 Service Unavailable and request a cool-down time (in seconds) using the Retry-After header. This causes CDStar to pause the consumer and not send any more requests for the requested cool-down period. If the Retry-After header is missing, the default retry.cooldown is used.

Any other response as well as connection problems or timeouts are logged as warnings and the request is sent again after retry.delay milliseconds. If a request fails more than retry.max times in a row, it is logged as an error and the consumer is paused for retry.cooldown milliseconds. This gives the consumer a chance to recover and also reduces logging noise considerably. Note that failing event are not discarded, but simply send again after the cool-down. Consumers MUST return a success status if they want to drop or ignore an event. Otherwise, they will receive the same event over and over again.

Slow consumers should queue and persist events locally and answer with 202 Accepted to prevent timeouts or events piling up too quickly. If a single request takes longer than http.timeout milliseconds, it is aborted and tried again. If the number of waiting events exceeds queue.size (per consumer), new events will be dropped and logged to a fail.log file.

The file configured with fail.log is used to store events that failed to be delivered. It contains one failed request per line, starting with the service URI, a single space, and the base64 encoded payload of the request. A timestamp is not logged since it can be easily recovered from the event payload itself.

The PushEventFilter only appends to this file and there is no automatic clean-up. A warning is logged if this file is not empty at service start-up time, but there is no automatic recovery or re-querying of events. This feature may be added in the future, though.

If you have consumers that are sensitive to lost events, make sure to check this file regularly. A short python script to re-submit events from a fail.log is shown here:

Events are buffered in an in-memory send-queue and re-queued on any errors. This helps to compensate short event bursts, temporary network failures or broker restarts.

Events that cannot be queued or re-queued are logged and dropped. This may happen during shutdown phase or when the send-queue overflows.

Events are not part of the transaction logic (yet). A forced shutdown or crash will loose all messages in the send-buffer. Also note that the broker itself may drop messages for various reasons, depending on its configuration. The possibility of loosing events MUST be considered when using this plugin.

This sink will buffer events in a bounded in-memory queue and sent them out one by one as fast as it can. Any errors (network or redis errors, buffer queue overflow) will cause events to be logged an dropped (WARN level). On shutdown, the sink tries its best to send all remaining events, but will only do so for a couple of seconds. On a crash, all queued events are lost.

Or in other words: This sink is NOT reliable in any way. Network errors or crashes will cause events to be lost. On the plus side, this sink will not slow down cdstar if the redis server fails.

The search gateway should accept POST requests at the configured target URL with Content-Type: application/json and return results in the same format as the CDSTAR v3 search API. Search queries will be sent as JSON documents with the following fields:

The q, fields, order, limit and scroll fields correspond to the (cleaned up) user provided search parameters as defined by the CDSTAR search API. vault and principal are added by CDSTAR. The search target should limit search results to entities visible to the specified principal. If no principal is present (null, missing or empty), the search should only return publicly visible results. If principal.privileged is true, the search should not filter by visibility and return all matching results.

Since the search gateway is not supposed to authenticate the searching user and trust the fields send by CDSTAR, it could be used to perform searches on behalf of another user, if accessed directly by an attacker. Make sure that the gateway is only reachable from the CDSTAR instance or is protected by HTTPS and some authentication mechanism (e.g. BASIC auth or secret headers).

No configuration necessary, but this plugin honors the global api.context setting (default: /). This may be required if the service path cannot be detected automatically and assets are not loaded correctly.

There is currently no configuration for this plugin. Uploads will be placed into ${path.var}/tus/.

The tus.io compatible REST endpoint is reachable under /tus at the root-level of the service (not /v3/tus but just /tus). After creating a TUS handle and uploading data following TUS protocol, the temporary file can be referenced as tus:<tusId>, where <tusId> is the last part of the TUS handle. For example, if your TUS handle was /tus/24e533e, then the internal reference to this resource would be tus:24e533e.

Currently only the Create Archive and Update Archive support server-side imports via the fetch:<target> functionality. For example, to import a completed TUS upload into an archive, you would send fetch:/path/to/target.file=tus:24e533e as a POST form parameter. Note that the digests must still be computed, so a fetch may take just as long as uploading the file directly. TUS usually does not improve overall throughput, but may improve reliability of large-file uploads over unreliable network connections. Use it wisely.

Incomplete TUS handles that do not see any new data will expire after 2 hours. Once complete, the TUS handle can be referenced for another 24 hours before it expires. Handles that are not needed anymore can (and should) be deleted faster with a single DELETE request to the TUS handle.

Storage objects are distributed into a directory tree with configurable depth, based on the first few character-pairs of the object ID. This reduces the maximum number of inodes per directory and helps keeping file system metadata cache-friendly, even for large pools with millions of objects. For a depth of d, the lookup path would be computed as follows: {poolName}/{id[0:2]}/…​/{id[(d-1)*2:d*2]}/{id}/. For example, given a default depth value of d=2, an object with ID 0123456789abcdef would be stored in myPool/01/23/0123456789abcdef/.

All files related to a specific pool object are stored in the same folder. Each object folder contains at least a HEAD symlink pointing to the latest {revision}.json index file. This file describes the state and content of the object in human readable form (json). There will be an extra index file for each revision of the object. Binary resources are stored in separate {sha256}.bin files. If object packing is enabled, some index or resource files may be bundled into packs and must be unpacked before they can be used (see below).

Each time an object is modified, a new {revision}.json index file is created and the HEAD symlink is updated. These files contain an utf-8 encoded JSON document describing the current state (contained resources, attributes and meta-data) of the storage object in a human-readable form.

Dates are stored as unix epoch timestamps with millisecond resolution (signed long integer). While not directly human readable, these are easily recognized and a very common exchange format for points in time. Most programming languages provide built-in tools to translate an epoch timestamp into a human readable form.

By default, the uncompressed binary content of non-empty resources are stored in the object directory as {sha256}.bin files named after the lower-case hex encoded sha256 digest of their content. These files always end in .bin regardless of their actual content-type. If this file is missing, the resource may either have been packed (see "Object Packing") or externalized (see "External resources") and additional steps are required to recover the binary content of the resource.

If the src field of a resource record is set, the corresponding {sha256}.bin resource file is subject to garbage-collection and may be removed at any time. In this case, the value of the src field should contain enough information to recover the resource file manually or with the help of an application-specific process. The src field MUST start with a prefix defined in this document, or with x- followed by an application defined location hint (e.g. an URI).

Resource files in an object directory may be bundled into one or more *.pack.zip files to save inodes and disk space. Compression can also help reducing IO pressure on the storage device in exchange for higher CPU usage during read access. This trade-of may be beneficial, in particular for rarely accessed objects or resources with highly compressible content.

Resources stored in a pack have a src value of pack:<pack-file-name> and follow default naming rules ({sha256}.bin) within the pack file.

NioPool may create temporary .tmp files or directories within an object directory. These may contain data required for recovery, so do not delete these files after an unclean shutdown or while the service is running. Temporary files that remain after an ordinary shutdown can be removed.

Any actor that creates or removes files other than *.tmp in an object directory or intends to change the target of the HEAD symlink MUST acquire a HEAD_NEXT file lock before doing so. The HEAD_NEXT file SHOULD be a symlink pointing to a (possibly not yet created) index file. To change the HEAD link, make sure that the HEAD_NEXT target exists and is synchronized to disk, then move-and-replace HEAD_NEXT to HEAD. Any error during this sequence should result in dangling HEAD_NEXT symlink protecting the object from further manipulation until manual or automatic recovery succeeded. In a disaster situation, either HEAD or HEAD_NEXT (or both) exists and the object can be rolled back or committed manually.

StoragePool configuration is stored by CDSTAR in a vault.yaml within the pool base directory and can be bootstrapped during vault creation with predefined parameters. NioPool supports the following configuration parameters:

Storage profiles can be either "hot" or "cold", which changes the way CDSTAR handles its local data.

Hot profiles causes CDSTAR to copy the archive content to external storage, but keep all data available in CDSTAR as well. While the profile is in effect, only administrative metadata (owner, ACLs, storage profile, …​) can be modified. The actual content (files and metadata) is write-protected to prevent stale LTS copies.

Cold profiles, on the other hand, allow CDSTAR to re-claim disk space by deleting archive files from disk after a copy was stored externally. Metadata is still kept available, but file content can no longer be accessed through CDSTAR. The profile needs to be changed to default or a hot profile to make file content available again.

Hot profiles are meant to increase long term availability or data integrity guarantees by storing important data in a second location. Cold profiles are mostly used to store large amounts of rarely accessed data in a more cost-effective way (e.g. on tape), while keeping meta-data search- and discoverable.

Profiles can be configured globally, and enabled or disabled per vault. They currently only have a name, and define a mode (hot or cold) and an associated LTS target, which is configured separately as a plugin. This allows multiple profiles to reference the same LTS target, but with different configuration.

Data migration from or to third party LTS systems is highly depended on the system in use. Multiple implementations are available and can be loaded via the CDSTAR plugin infrastructure. CDSTAR bundles a general purpose implementation that exports to BagIt directories and allows an external process to perform the actual LTS migration asynchronously.

LTS handlers are referenced by name, so special care must be taken when removing or renaming LTS handlers. Do not remove or rename an LTS target as long as there are archives still referencing it.

Moving data out of the CDSTAR system, especially with cold profiles, bears some risks that should be well understood before enabling the LTS feature. Please read this chapter carefully.

After a successful migration to an LTS target, CDSTAR stores the LTS name and a unique location identifier (generated by the LTS) into non-public archive properties. These are used to recover missing files in case of a future profile change. Cold profiles allow CDSTAR to remove local copies of archived files after successfully copying these files to LTS. If the LTS goes away, for whatever reason, then CDSTAR has no way to recover missing files and the archive is stuck in cold state. File content will be unavailable and data migration after profile changes will fail.

This LTS target exports archives into BagIt folders, and is designed to work with external worker processes for the actual migration from/to LTS storage (e.g. tape).

The exporter will create a BagIt package in a temporary folder, then rename it to [name].bagit with a unique name. A worker process may check for these folders and copy or move data to LTS.

The importer will create a file named [name].want and start the import as soon as the [name].bagit folder can be found. A worker process should check for these [name].want files and recover the missing [name].bagit folder from LTS. Once complete, the importer will delete the [name].want file and the recovered [name].bagit folder can be cleaned up by the worker.

If the external copy is no longer needed, a [name].delete file is created. A worker process should watch for these files, remove the external copy (if any), remove the [name].bagit directory (if present), and then also remove the [name].delete file.

External workers are allowed to create additional files for their own state handling, as long as they do not interfere with the names defined here.

On storage level, snapshots live in separate storage objects, but are created in a way that allows them to share common data files with their source archive or other snapshots, if supported by the storage back-end. This ensures that snapshots only take up a minimum amount of additional storage space and are usually way more efficient than actually copying an entire archive. NioPool implements this on file-system level by hard-linking files with the same content, and only creating a copy if content changes (copy on write semantics).