Destination Send Workflow

This vignette explains the internal workflow for sending content to remote destinations during builds. It describes the three main stages that occur when projr uploads artifacts to GitHub releases, local directories, or OSF nodes.

Overview

When you run a production build (projr_build_patch(), projr_build_minor(), or projr_build_major()), projr can automatically send artifacts to configured remote destinations. This process is controlled by the .dest_send_label() function in R/dest-send-label.R, which orchestrates three distinct stages:

Getting the Remotes - Resolve remote locations and metadata
Building the Plan - Determine which files need to be added, removed, or updated
Implementing the Plan - Execute the file operations and update remote metadata

Each stage builds upon the previous one to ensure efficient and accurate synchronization between your local project and remote destinations.

Stage 1: Getting the Remotes

The first stage resolves remote locations and determines what versions exist on the remote destination.

flowchart TD
    Start([Start: .dsl_get_remotes]) --> GetPre[Get remote_pre<br/>Base remote location]
    GetPre --> GetDestFull[Get remote_dest_full<br/>Full version destination]
    GetDestFull --> GetDestEmpty[Get remote_dest_empty<br/>Empty version destination]
    GetDestEmpty --> GetVersionComp[Determine version_comp<br/>Version to compare against]
    
    GetVersionComp --> CheckInspect{inspect == 'none'?}
    CheckInspect -->|Yes| ReturnNull[version_comp = NULL]
    CheckInspect -->|No| CheckStructure{structure?}
    
    CheckStructure -->|latest| LatestLogic[Check if remote exists<br/>Trust based on manifest]
    CheckStructure -->|archive| ArchiveLogic[Find latest acceptable<br/>archive version]
    
    LatestLogic --> GetComp[Get remote_comp<br/>Remote for comparison]
    ArchiveLogic --> GetComp
    ReturnNull --> GetComp
    
    GetComp --> End([Return remote list])
    
    style Start fill:#e1f5e1
    style End fill:#e1f5e1
    style GetVersionComp fill:#fff4e1

Key Outputs:

remote_pre - The base remote location (e.g., GitHub release tag, local directory path)
remote_dest_full - Where files will be uploaded (full version, not empty)
remote_dest_empty - Empty version variant (for GitHub empty placeholders)
version_comp - Version to compare against (NULL if no trusted version available)
remote_comp - The actual remote object used for comparison

Version Comparison Logic:

The version_comp determines which remote version can be trusted for comparisons:

NULL - No trusted version available; treat remote as empty (upload everything)
Version string - Trusted version exists; compare local files against this version

For latest structure: - Uses the current remote contents if they exist - Trusts manifest if available and valid

For archive structure: - Finds the latest archive version that matches local state - Must be recent enough (no local changes since that version) - For inspect = "file", always trusts the version by hashing files - For inspect = "manifest", only trusts if manifest is valid

Stage 2: Building the Plan

The second stage compares local and remote state to determine which files need to be uploaded, removed, or updated.

flowchart TD
    Start([Start: .dsl_get_plan]) --> GetFilenames[Get file lists<br/>fn_source & fn_dest]
    
    GetFilenames --> Strategy{send_strategy?}
    
    Strategy -->|upload-all| UploadAll[fn_add = all local files<br/>fn_rm = none]
    Strategy -->|upload-missing| UploadMissing[fn_add = local files not on remote<br/>fn_rm = none]
    Strategy -->|sync-diff| SyncDiff[Compare hashes<br/>fn_add = new + changed<br/>fn_rm = deleted]
    Strategy -->|sync-purge| SyncPurge[Compare hashes<br/>fn_add = all local<br/>purge = true]
    
    UploadAll --> BuildActions[Translate to actions]
    UploadMissing --> BuildActions
    SyncDiff --> BuildActions
    SyncPurge --> BuildActions
    
    BuildActions --> CheckCue{send_cue?}
    CheckCue -->|never| SkipSend[Skip upload]
    CheckCue -->|always| ProceedSend[Proceed with upload]
    CheckCue -->|if-change| CheckChanges{Any changes?}
    
    CheckChanges -->|Yes| ProceedSend
    CheckChanges -->|No| SkipSend
    
    ProceedSend --> BuildMetadata[Build metadata updates<br/>- manifest.csv<br/>- VERSION file<br/>- changelog]
    SkipSend --> End
    BuildMetadata --> End([Return plan])
    
    style Start fill:#e1f5e1
    style End fill:#e1f5e1
    style BuildActions fill:#fff4e1

Key Outputs:

fn_add - Files to upload to the remote
fn_rm - Files to remove from the remote
version - Updated VERSION file content
manifest - Updated manifest.csv content
purge - Whether to purge all remote files before upload
ensure_remote_dest_exists - Whether to create the remote if it doesn’t exist
is_remote_dest_empty - Whether the remote will be empty after operations

Send Strategies:

Different strategies determine how local and remote files are compared:

upload-all - Upload all local files, ignore remote state
upload-missing - Only upload files that don’t exist on remote
sync-diff - Upload new/changed files, remove deleted files
sync-purge - Remove all remote files, then upload all local files

Inspection Modes:

The inspect parameter controls how projr determines what’s on the remote:

manifest - Use remote manifest.csv (fast, requires valid manifest)
file - Download and hash actual files (slower, always accurate)
none - Treat remote as empty (always upload everything)

Stage 3: Implementing the Plan

The third stage executes the file operations and updates remote metadata.

flowchart TD
    Start([Start: .dsl_implement_plan]) --> CheckPurge{purge?}
    
    CheckPurge -->|Yes| PurgeRemote[Remove all files<br/>from remote_dest_full]
    CheckPurge -->|No| AddFiles
    
    PurgeRemote --> AddFiles[Add files from fn_add]
    
    AddFiles --> CheckAdd{fn_add empty?}
    CheckAdd -->|No| CreateIfNeeded{remote exists?}
    CheckAdd -->|Yes| RemoveFiles
    
    CreateIfNeeded -->|No| CreateRemote[Create remote destination]
    CreateIfNeeded -->|Yes| UploadFiles[Upload files to remote]
    
    CreateRemote --> UploadFiles
    UploadFiles --> RemoveFiles[Remove files from fn_rm]
    
    RemoveFiles --> CheckRm{fn_rm empty?}
    CheckRm -->|No| DeleteFiles[Delete files from remote]
    CheckRm -->|Yes| FinalizeRemotes
    
    DeleteFiles --> FinalizeRemotes[Finalize remote states<br/>- Remove empty variant if full exists<br/>- Create empty variant if needed]
    
    FinalizeRemotes --> CheckBoth{Both full & empty exist?}
    CheckBoth -->|Yes| RemoveEmpty[Remove empty remote]
    CheckBoth -->|No| CheckCreate{Create empty needed?}
    
    RemoveEmpty --> UpdateMetadata
    CheckCreate -->|Yes| CreateEmpty[Create empty remote]
    CheckCreate -->|No| UpdateMetadata
    
    CreateEmpty --> UpdateMetadata[Upload metadata files<br/>- manifest.csv<br/>- VERSION file<br/>- CHANGELOG.md]
    
    UpdateMetadata --> End([Complete])
    
    style Start fill:#e1f5e1
    style End fill:#e1f5e1
    style FinalizeRemotes fill:#fff4e1

Key Operations:

Purge (if purge = TRUE) - Remove all existing files from remote
Add files - Upload files in fn_add to remote destination
Remove files - Delete files in fn_rm from remote destination
Finalize remotes - Handle empty/full remote variants:
- For GitHub: manages -empty.zip placeholder files
- For local: manages empty version directories
- Ensures only one variant exists (full takes priority)
Update metadata - Upload manifest.csv, VERSION file, and optionally CHANGELOG.md

Remote Variants:

For archive structures, projr manages two variants of each version:

Full remote (remote_dest_full) - Contains actual files (e.g., output-v0.0.1.zip)
Empty remote (remote_dest_empty) - Placeholder when no files exist (e.g., output-v0.0.1-empty.zip)

This allows projr to distinguish between: - “Version never uploaded” (neither variant exists) - “Version uploaded but empty” (empty variant exists) - “Version uploaded with files” (full variant exists)

Only one variant exists at a time. When files are added to an empty remote, the empty variant is deleted and the full variant is created.

Complete Workflow

Here’s how all three stages work together during a build:

flowchart TD
    Start([Production Build]) --> Loop{For each<br/>destination}
    
    Loop --> Stage1[Stage 1: Get Remotes<br/>.dsl_get_remotes]
    
    Stage1 --> Stage2[Stage 2: Build Plan<br/>.dsl_get_plan]
    
    Stage2 --> CheckCue{send_cue<br/>allows?}
    CheckCue -->|No| Loop
    CheckCue -->|Yes| Stage3[Stage 3: Implement Plan<br/>.dsl_implement_plan]
    
    Stage3 --> Loop
    Loop --> End([Build Complete])
    
    style Start fill:#e1f5e1
    style End fill:#e1f5e1
    style Stage1 fill:#e3f2fd
    style Stage2 fill:#fff9c4
    style Stage3 fill:#f3e5f5

Build Integration:

The destination send process is triggered during post-build operations:

Pre-build - Clear directories, hash input files
Build - Render documents and scripts
Post-build - Hash output files, commit to git
Destination Send - Upload to remotes (this workflow)
Dev Version Bump - Increment to next dev version

For each configured destination in _projr.yml: - Check if send_cue condition is met (always, if-change, never) - Get remotes and determine comparison version - Build upload/removal plan based on send_strategy and send_inspect - Execute plan if approved by cue logic

Configuration Parameters

The workflow behavior is controlled by parameters in _projr.yml:

structure (where versions are stored): - archive - Create separate versions (v0.0.1, v0.0.2, etc.) - latest - Overwrite same location each time

send_cue (when to upload): - always - Upload every build - if-change - Only upload if content changed - never - Skip upload

send_strategy (how to upload): - sync-diff - Upload changed/new files, remove deleted files - sync-purge - Remove all, then upload all - upload-all - Upload all files - upload-missing - Only upload missing files

send_inspect (how to check remote): - manifest - Use manifest.csv (fast) - file - Hash actual files (slower, accurate) - none - Treat as empty (always upload)

Examples

Example 1: First Upload (No Remote Exists)

Configuration:

structure: archive
send_cue: if-change
send_strategy: sync-diff
send_inspect: manifest

Workflow: 1. Get Remotes: version_comp = NULL (no remote exists) 2. Build Plan: fn_add = all local files, fn_rm = [] 3. Implement: Create remote, upload all files, write manifest

Example 2: Incremental Update (Files Changed)

Configuration:

structure: archive
send_cue: if-change
send_strategy: sync-diff
send_inspect: manifest

Workflow: 1. Get Remotes: version_comp = "0.0.1" (last upload version) 2. Build Plan: Compare against v0.0.1 manifest - fn_add = [new_file.txt, changed_file.txt] - fn_rm = [deleted_file.txt] 3. Implement: Upload 2 files, remove 1 file, update manifest

Example 3: No Changes (Skip Upload)

Configuration:

structure: archive
send_cue: if-change
send_strategy: sync-diff
send_inspect: manifest

Workflow: 1. Get Remotes: version_comp = "0.0.1" 2. Build Plan: Compare against v0.0.1 manifest - fn_add = [], fn_rm = [] (no changes) 3. Implement: Skip upload (cue not met)

Example 4: Latest Structure (Always Overwrite)

Configuration:

structure: latest
send_cue: always
send_strategy: sync-purge
send_inspect: none

Workflow: 1. Get Remotes: version_comp = NULL (inspect = none) 2. Build Plan: purge = true, fn_add = all local files 3. Implement: Remove all remote files, upload all local files

Trust and Manifest Validation

The workflow maintains a trust system to ensure accurate synchronization:

Trusted Versions (can use manifest for comparison): - Uploaded with sync-diff or sync-purge strategy - Has valid manifest.csv on remote - No local changes since that version

Untrusted Versions (must hash actual files): - Uploaded with upload-all or upload-missing strategy - No manifest.csv on remote - Manifest doesn’t match local manifest for that version

The VERSION file tracks trust status with an asterisk:

project: 0.0.2
output: 0.0.1*  # Asterisk means untrusted
raw-data: 0.0.1  # No asterisk means trusted

When a version is untrusted, projr falls back to hashing actual files (inspect = "file") even when inspect = "manifest" is configured.

Performance Considerations

Fast Configuration (recommended default):

structure: archive
send_cue: if-change
send_strategy: sync-diff
send_inspect: manifest

This is fastest because: - Archives preserve version history - if-change skips unchanged content - sync-diff uploads only differences - manifest avoids downloading files

Simple Configuration (easier to understand):

structure: latest
send_cue: always
send_strategy: upload-all
send_inspect: none

This is simpler but slower: - Always uploads all files - No comparison logic needed - No manifest dependency

Debugging

Enable debug output to see workflow details:

# Set debug output level
Sys.setenv(PROJR_OUTPUT_LEVEL = "debug")

# Run production build
projr_build_patch()

# View detailed log
projr_log_view()

Debug output shows: - Remote configuration (id, structure, strategy, inspect) - Remote existence checks - Upload plan (files to add/remove) - File operation details - Metadata updates

Overview

Stage 1: Getting the Remotes

Stage 2: Building the Plan

Stage 3: Implementing the Plan

Complete Workflow

Configuration Parameters

Examples

Example 1: First Upload (No Remote Exists)

Example 2: Incremental Update (Files Changed)

Example 3: No Changes (Skip Upload)

Example 4: Latest Structure (Always Overwrite)

Trust and Manifest Validation

Performance Considerations

Debugging

See Also