Understanding and Responding to Job and Model Failures

When a Hover job or an individual model within a job can't be successfully processed by our reconstruction pipeline, its state updates to failed. Hover then dispatches webhook notifications to let your integration know about these status changes.

It's important to note that a job's state is independent of its models' states. A job might even reach a completed state if one or more of its constituent models have failed to reconstruct. So, monitoring for specific failures at both the job and model level is crucial.

Getting the specific failure_reason after a job or model has failed is essential for giving meaningful feedback to your users or kicking off automated fixes within your system. This lets you go beyond a simple "failed" status and offer actionable insights.

Recommended Approach to Retrieve Failure Reasons

To make sure you capture all relevant processing failures and their specific reasons, we recommend monitoring for both job-state-changed-v2 and model-state-changed webhook events.

Steps to Retrieve Failure Reasons

1. Receive Webhook Notification:
   Your integration should be set up to receive:
   1. job-state-changed-v2 webhooks: Monitor these for instances where the state of the overall job transitions to "failed".
   2. model-state-changed webhooks: Specifically monitor these for individual models where their state transitions to "failed".
   3. The webhook payload for both event types will include the job_id (and model_id for model-state-changed events) of the affected entity.
2. Query the Show Job Details Endpoint:
   Regardless of whether a job or a model failed, use the job_id and/or model_id from the received webhook payload to make a GET request to the Show Job Details endpoint. This endpoint gives you comprehensive information about the job, including details for all associated models and their respective states and failure reasons.
3. Interpret the failure_reason:
   The response from the Show Job Details endpoint will contain the full job object.
   1. For Job-Level Failures: If the job_state is "failed", the top-level failure_reason field within the job object will contain the reason for the overall job's failure.
   2. For Model-Level Failures: If a specific model's state within the job object (found inside the models array) is "failed", that individual model object will have its own failure_reason field, explaining why that particular model couldn't be reconstructed.
   3. Actionable Insight: Use the relevant failure_reason (either job-level or model-level) to inform your customers directly, prompt them to take specific remediating actions or log the issue for internal review. Giving granular reasons helps users understand exactly what went wrong and how to fix it.

Possible Failure Reasons

Below is a comprehesive list of the possible failure reasons that you might see on either the job or model level: