deco - Notice history

All systems operational

deco - Operational

100% - uptime
Mar 2024 · 100.0%Apr · 100.0%May · 99.98%

Previous page

Next page

Mar 2024
Apr 2024
May 2024

deno - Operational

VTEX - Operational

Third Party: Amazon Web Services (AWS) → AWS ec2-sa-east-1 - Operational

Third Party: Cloudflare → Cloudflare Sites and Services → API - Operational

Third Party: Cloudflare → Cloudflare Sites and Services → CDN/Cache - Operational

Third Party: Cloudflare → Cloudflare Sites and Services → DNS Root Servers - Operational

Third Party: Cloudflare → Cloudflare Sites and Services → SSL Certificate Provisioning - Operational

Third Party: Cloudflare → Cloudflare Sites and Services → Workers - Operational

Third Party: Fly.io → AWS s3-sa-east-1 - Operational

Third Party: Fly.io → Application Hosting → GRU - São Paulo, Brazil - Operational

Third Party: Github → Actions - Operational

Third Party: Github → API Requests - Operational

Third Party: Github → Git Operations - Operational

Third Party: Github → Pull Requests - Operational

Third Party: Github → Webhooks - Operational

Third Party: Shopify → Shopify API & Mobile - Operational

Third Party: Shopify → Shopify Checkout - Operational

Third Party: Shopify → Shopify Point of Sale - Operational

Third Party: Supabase → Supabase Cloud → AWS ec2-sa-east-1 - Operational

Third Party: Discord → Voice → Brazil - Operational

Third Party: Fly.io → Edge Proxy → Routing - Operational

Third Party: Supabase → Auth - Operational

Third Party: Supabase → Storage - Operational

Notice history

May 2024

Admin site outage
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    A redeployment of the previous admin version was done, and the system is now operational.

  • Identified
    Identified

    The deployment of a new admin version with issues.

  • Investigating
    Investigating
    We are currently investigating this incident.

Apr 2024

Admin site unstable (1016 origin DNS error)
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Identified
    Identified

    Following an infrastructure upgrade to our Kubernetes (k8) system, the ELB DNS has been modified. Consequently, we've observed occurrences of error 1016 (CF DNS origin error). We are actively addressing this issue and appreciate your patience as we resolve it.

  • Investigating
    Investigating
    We are currently investigating this incident.
GitHub Account Flagged
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We are pleased to inform you that the issue with our GitHub organization being flagged has been fully resolved.

    Visibility and access to our repositories have been restored.

    We sincerely appreciate your patience and understanding while we worked to resolve this matter.

    We are monitoring now.

  • Investigating
    Investigating

    Dear deco.cx Users,

    We regret to inform you that our GitHub organization has been flagged, resulting in a temporary loss of visibility and access to our repositories. This issue affects our ability to provide services and support through GitHub.

    We apologize for any inconvenience this may cause and appreciate your patience and understanding during this time.

    Sincerely,
    The deco.cx Team

Mar 2024

Partial outage on specific hostnames
  • Resolved
    Resolved
    This incident has been resolved.
  • Investigating
    Investigating

    We are currently investigating an incident with SSL handshake failure in some deco sites.

Elevated number of 403 from VTEX
  • Resolved
    Resolved

  • Monitoring
    Monitoring

    Our infrastructure provider blocked the problematic IP from being associated with our machines, and the issues related to this query ceased.

  • Update
    Update

    VTEX's team responds, confirming that only one IP was blocked and had been since 2021. However, this specific IP had started generating traffic volume from March 6th, coinciding with the start of the incident.

    VTEX informs us of the blocked IP and indicates that action could be taken on Monday.

    We contacted our infrastructure provider requesting the rotation of this IP and its exclusion from allocation to our sites.

  • Update
    Update

    We are continuing to work on a fix for this incident.First contact made with VTEX, accompanied by all collected evidence.

    Throughout the day, manual interventions were carried out on the stores to reduce the number of errors, such as redeployments, blocking potential bots, and reducing calls to VTEX. Efforts were also made to enhance error logging to pinpoint the root cause.

  • Identified
    Identified

    The team observes a recurrence of the error across different stores, with the common denominator being the integration with VTEX. Initial suspicions were towards VTEX throttling, as not all VTEX customers were affected, and those who were did not experience it 100% of the time.

  • Investigating
    Investigating

    We encountered an issue with certain components not displaying on searchResults/Category pages on one of our sites. After a redeploy, the problem was resolved. However, we have since noticed a growing number of 403 calls from VTEX."

Mar 2024 to May 2024

Next