Cognyzer Cognyzer Terminus-2.0 Docs
06Section 02 · Submission Process

Platform Submission

Platform Submission Guide

Complete step-by-step guide for creating and submitting tasks through the Cognyzer Expert Platform.

High-Level Workflow

Tasking is performed through the Terminus-2nd-Edition project on the Cognyzer Expert Platform. The complete workflow:

  1. Download the correct task skeleton template. There are 3!
  2. Extract and rename the folder
  3. Write your task instructions and configure metadata
  4. Set up the Docker environment
  5. Create and test your solution
  6. Write and verify tests
  7. Run agents
  8. Create ZIP file, check rubric generation checkbox, and Submit on platform
  9. Review CI feedback and iterate + review generated rubric and edit for accuracy and completeness
  10. When CI is passing, submit on platform to a reviewer

Step 1: Download the Correct Task Skeleton

Choose and download the appropriate task skeleton template for your task type:

There are 3 task skeletons to choose from: - "Regular" Task Skeleton: Use for all non-UI-Building and non-Milestone tasks - UI Task Skeleton: Use for all UI-Building subtype tasks - Milestone Task Skeleton: Use for all tasks which contain milestones

Step 2: Extract and Rename

  1. Extract the ZIP file to your desired location
  2. Rename the folder to match your task name (use kebab-case, e.g., fix-memory-leak-python)

Step 3: Write Task Instructions and Configuration

Author your instruction.md

See the Prompt Styling Guide for detailed instructions and requirements about how you should style your instructions.

Configure task.toml

Set up metadata and configuration:

# Task configuration schema version
version = "2.0"

# Task metadata (author, difficulty, categorization)
[metadata]
author_name = "anonymous"
author_email = "anonymous"
difficulty = "unknown"
category = "software-engineering"
# Options for subcategories are: "long_context", "tool_specific", "api_integration", "db_interaction", "ui_building"
subcategories =
# The number of milestones in the task (can be zero if not a milestone task)
number_of_milestones = 0
# Size of the codebase: minimal -> 0-20 files, small -> 20+ files, large -> 200+ files. Include all files in the environment when counting.
codebase_size = "minimal"
# Coding languages used in the oracle solution or required by the agent
languages = [ "bash" ]
# For tool_specific, api_integration, and db_interaction subcategories, please include specific tool, api framework, or database software
tags = [ "file-operations",]
# Estimated time to complete (minutes)
expert_time_estimate_min = 60
junior_time_estimate_min = 120

# Verifier: runs the test script to check task completion
[verifier]
timeout_sec = 450.0

# Agent: limits how long the agent can run when attempting the task
[agent]
timeout_sec = 900.0

# Sandbox environment limits
[environment]
build_timeout_sec = 600.0
cpus = 2
memory_mb = 4096
storage_mb = 10240

Step 4: Configure Docker Environment

Edit the environment/Dockerfile to set up your task environment:

  • Add any dependencies required by your task
  • Pin all package versions for reproducibility
  • Never copy solution/ or tests/ folders in the Dockerfile

For multi-container environments or custom configurations, see the Docker environment documentation.

Docker Troubleshooting

If you encounter Docker issues:

  1. Ensure Docker Desktop is running
  2. On macOS, enable in Advanced Settings: "Allow the default Docker socket to be used (requires password)"
  3. Try these commands if needed:
sudo dscl . create /Groups/docker
sudo dseditgroup -o edit -a $USER -t user docker

Step 5: Test Your Solution Locally

Enter your task container interactively to test your solution:

harbor tasks start-env -p <task-folder> -i

While in the container, test your solution approach to ensure it works as expected.

Step 6: Create Solution File

Create solution/solve.sh with the verified commands:

  • This file is used by the Oracle agent to verify the task is solvable
  • Must be deterministic (same result every run)
  • Should demonstrate the command sequence, not just output the answer
  • For milestone tasks: also create solution/solve1.sh ... solution/solveN.sh; each must be independently scoped to milestone X, and solve.sh must chain them in order

See Writing Oracle Solution for guidance.

Step 7: Write Tests

Create tests/test.sh and test files to verify task completion:

  • The test script must produce /logs/verifier/reward.txt
  • Create pytest unit tests in tests/test_outputs.py (or similar)
  • Place any test dependencies in the tests/ directory
  • For milestone tasks: create tests/test_m1.py ... tests/test_mN.py, and ensure each test_mX.py scores only milestone X completion

Tests must fully cover the prompt: explicit requirements, implicitly expected behavior, and critical edge cases - with every prompt requirement mapped to a test. See Writing Tests for detail.

Example test.sh:

#!/bin/bash

# Install curl
apt-get update
apt-get install -y curl

# Install uv
curl -LsSf https://astral.sh/uv/0.9.5/install.sh | sh

source $HOME/.local/bin/env

# Check if we're in a valid working directory
if [ "$PWD" = "/" ]; then
    echo "Error: No working directory set. Please set a WORKDIR in your Dockerfile before running this script."
    exit 1
fi

# Don't change anything below this line
uvx \
  -p 3.13 \
  -w pytest==8.4.1 \
  -w pytest-json-ctrf==0.3.5 \
  pytest --ctrf /logs/verifier/ctrf.json /tests/test_outputs.py -rA

if [ $? -eq 0 ]; then
  echo 1 > /logs/verifier/reward.txt
else
  echo 0 > /logs/verifier/reward.txt
fi

See Writing Tests for detailed guidance.

Step 8: Run Oracle Agent

Verify your solution passes all tests:

harbor run -a oracle -p <task-folder>

This should PASS. If it doesn't, fix issues before proceeding.

Step 9: Test with Real Agents

  1. Set up your API key (received via email):
export OPENAI_API_KEY=<your-portkey-api-key>
export OPENAI_BASE_URL=https://api.portkey.ai/v1
  1. Run with GPT-5:
harbor run -a terminus-2 -m openai/@openai/gpt-5.2 -p <task-folder>
  1. Run with Claude Opus 4.6:
harbor run -a terminus-2 -m anthropic/@anthropic/claude-opus-4-6 -p <task-folder>

Run each agent 2-3 times to gauge pass rate. Your task should have < 80% pass rate to be accepted.

Step 10: Run LLMaJ Checks Locally

Run LLMaJ checks before submitting:

GPT-5:

harbor run -a terminus-2 -m openai/@openai/gpt-5.2 -p <task-folder>

Claude Opus 4.6:

harbor run -a terminus-2 -m anthropic/@anthropic/claude-opus-4-6 -p <task-folder>

All checks should pass before submission.

Step 11: Final Verification

Before submitting, verify:

  • Oracle agent passes
  • All LLMaJ checks pass
  • Tested against real agents (pass rate < 80%)
  • All files are present and correct

Run final checks:

# Oracle agent
harbor run -a oracle -p <task-folder>

# LLMaJ checks
harbor tasks check -m openai/@openai/gpt-5.2 harbor_tasks/<task_name>

Step 12: Create ZIP File

Important: Select the individual files inside your task folder, not the folder itself.

.
├── instruction.md   ← Select these
├── task.toml       ←
├── environment/     ←
│   ├── Dockerfile   ← (or docker-compose.yaml)
│   └── [build files]
├── solution/        ←
│   └── solve.sh
└── tests/           ←
    ├── test.sh
    └── [test files]

On macOS: 1. Open the task folder 2. Select all files (Cmd+A) 3. Right-click → Compress

On Windows: 1. Open the task folder 2. Select all files (Ctrl+A) 3. Right-click → Send to → Compressed folder

Step 13: Submit to Platform

  1. Go to the Cognyzer Expert Platform
  2. Navigate to Terminus-2nd-Edition
  3. Click Start on the Submission node
  4. Upload your ZIP file
  5. Keep "Send to reviewer" unchecked
  6. Check the rubrics checkbox
  7. Submit

Step 14: Check CI results and rubric then Iterate until CI looks good

  1. After email notification that your submission is now back in your revision queue, go to Cognyzer Expert Platform.
  2. Find your task on homescreen
  3. Click "Revise"
  4. Check CI Results & update task as needed
  5. Check the now generated rubric and edit it within the textbox for accuracy and completeness
  6. Re-upload a new .zip file if necessary - Use the Reviewer Checklist to confirm you have addressed high-severity review criteria before resubmitting. - If you make any significant changes to your task, you must update your rubric accordingly in order to align with the current version of your task.
  7. Keep "Send to Reviewer" Unchecked
  8. Submit

Step 15: Submit your task to Reviewer

  1. After email notification that your submission is now back in your revision queue, go to Cognyzer Expert Platform.
  2. Find your task on homescreen
  3. Click "Revise"
  4. Check CI results
  5. Check rubric and edit for accuracy and completeness
  6. If all good, check "Send to Reviewer"
  7. Submit

Step 16: Monitor Status

After submission. wait for peer review (1-3 business days)

After Submission

Review Process

  1. Automated checks runs immediately
  2. Peer review within 1-3 business days
  3. Feedback provided if changes needed
  4. Acceptance when all criteria met

If Changes Requested

  1. Review the feedback carefully
  2. Make requested changes locally
  3. Re-run all checks
  4. Create new ZIP and resubmit

Common Issues

ZIP Structure Wrong

Problem: Files nested in extra folder.

Fix: ZIP the files directly, not the containing folder.

Missing Files

Problem: Forgot to include a file.

Fix: Verify all files are in ZIP before uploading. Check that instruction.md, task.toml, environment/, solution/, and tests/ are all included.

  • If milestones in task, milestones.md must be included in the root of the .zip.
  • If milestones in task, include solution/solve1.sh ... solution/solveN.sh and tests/test_m1.py ... tests/test_mN.py.

CI Failures After Upload

Problem: Local checks passed but platform CI fails.

Fix: Check for environment differences, re-run locally with exact CI commands.

Docker Build Fails

Problem: Docker build fails on platform but works locally.

Fix: Ensure all dependencies are pinned to specific versions, including the Docker base image (and any images in docker-compose.yaml). Check for platform-specific differences.

Video Walkthroughs

Watch these step-by-step videos to learn how to create and test your task:

1. Running Your Task

2. Creating a solution/solve.sh

The video refers to solution/solve.sh file. Using Harbor, this refers to solution/solve.sh

3. Creating Tests for Your Task

Next Steps

  • Understand task components
  • Review the submission checklist
  • Reviewer Checklist
  • Learn about CI checks
  • Understand the review process