Troubleshooting

Start here when an upload, download, login, or background storage operation fails. Check health first, then narrow the problem by wallet, cache, task, or storage provider signals.

First Checks

curl http://127.0.0.1:9090/healthz
synaps3 admin status
synaps3 admin task stats

Expected healthy baseline:

{"status":"ok"}

If health is not ok, use the error text as the next branch.

Setup Mode

Health may return:

{"status":"setup"}

This means SynapS3 is missing required settings. Review the configuration validation details before changing a value.

Check the reported fields:

synaps3 admin status
synaps3 admin settings get

If the wallet private key is missing, generate one:

synaps3 wallet generate

If the missing value is the wallet key, set the generated private key in SYNAPS3_FILECOIN_PRIVATE_KEY or filecoin.private_key in the config file. Restart SynapS3, check /healthz, and verify the effective settings.

Expected result: health changes from setup to ok after restart.

Unhealthy Background Tasks

Example:

{"status":"unhealthy","errors":["worker/tasks: not responding"]}

Check task pressure:

synaps3 admin task stats
synaps3 admin task list --status running --limit 20

After a restart, unfinished tasks become eligible to continue. If background task processing stays unhealthy, record the current task state, inspect logs, and then restart the service.

Wallet Funding or Deposit Fails

Check wallet status:

synaps3 admin status

For Calibration, fund the wallet address again:

synaps3 wallet fund-testnet 0x...

Then retry deposit and FWSS approval:

synaps3 wallet deposit 2 # 2 USDFC
synaps3 wallet approve

If faucet funding fails, claim manually from ChainSafe or Plumbline, then rerun synaps3 admin status.

Successful faucet claims print CalibnetUSDFC: <hash> and CalibnetFIL: <hash>. A confirmed deposit or approval prints Transaction: <hash> and Status: confirmed; an existing approval prints FWSS approval: already approved. If these results do not appear, verify RPC connectivity, wallet funds, and the reported error before retrying.

Cache Full

Upload endpoints can fail when cache capacity is exhausted. Check usage:

synaps3 admin status
synaps3 admin settings get cache.max_size_gb
synaps3 admin settings get cache.eviction_policy
synaps3 admin settings get cache.lru_high_watermark_percent
synaps3 admin settings get cache.lru_low_watermark_percent

Recovery options:

• Confirm the host has free disk space, then increase cache.max_size_gb if capacity allows.
• Restore storage provider connectivity and background task progress so queued uploads can complete and cache eviction can run.
• Use the default lru policy for capacity-based cleanup. Lower the high watermark to leave more write headroom, and keep 0 <= low < high <= 100.
• Use after_upload only when each version should be removed asynchronously after its bucket's minimum durable copies commit.
• Use none when automatic removal must be disabled.

LRU cannot remove multipart staging data, versions below their bucket's minimum durable copies, or versions without a readable committed remote copy. A write does not synchronously run cleanup, so 507 Insufficient Storage can continue until background cleanup catches up or safe candidates become available.

Failed LRU deletion tasks remain visible as failed work. Fix the reported filesystem or database problem first; use synaps3 admin task retry <id> when the task is marked retryable.

After changing a cache setting, restart SynapS3, check /healthz, and verify the effective cache values with synaps3 admin settings get.

Failed Tasks

List failed work:

synaps3 admin task list --status failed --limit 100

Retry only after RPC connectivity, storage provider availability, wallet funds, FWSS approval, and cache disk capacity are ready.

synaps3 admin task retry 42

The API indicates whether Retry is available for each failed task. Provider replacement work is recovered from Details → Storage → Data Sets. A wallet operation can be recovered from Tasks only when no broadcast started; an uncertain broadcast remains non-retryable. An uncertain Store offers Check again, which observes the provider without uploading again. For remote-copy removal that remains unconfirmed after 24 hours, Recover checks the chain first and may submit another paid request if the copy is still present and not queued; the earlier request may still succeed. If the data set is no longer active, the chain cannot confirm this individual removal, so the task keeps checking instead of claiming the copy is gone. Use Dismiss or synaps3 admin task acknowledge <id> only after reviewing the failure; acknowledged tasks remain available for the configured retention period before cleanup. When failures have piled up, Dismiss all on the Tasks page clears the ones the current Operation filter selects, and synaps3 admin task acknowledge --type <operation> --yes does the same from the CLI; failures recorded after you confirm stay in the list.

Provider or RPC Issues

Check provider health and Filecoin readiness in the dashboard, or inspect the Admin API:

curl -u admin http://127.0.0.1:9090/api/v1/filecoin/readiness
curl -u admin http://127.0.0.1:9090/api/v1/observability/providers

Enter the Admin password at curl's no-echo prompt.

Recovery:

• Restore the configured filecoin.rpc_url.
• Confirm provider URLs are reachable from the SynapS3 host.
• Keep filecoin.allow_private_networks = false unless private provider URLs are expected and trusted.

S3 Client Cannot Upload

Check these in order:

1. S3 client uses path-style addressing.
2. Access key and secret came from synaps3 admin s3-user create.
3. Endpoint is http://localhost:8080 for local evaluation or the correct HTTPS address for production.
4. Object size is between 127 and 1,065,353,216 bytes, and the object key meets the S3 compatibility limits.
5. Dashboard task view shows whether Filecoin storage is queued, running, waiting, or failed.