deco - Notice history

This incident has been resolved.

A redeployment of the previous admin version was done, and the system is now operational.

The deployment of a new admin version with issues.

Following an infrastructure upgrade to our Kubernetes (k8) system, the ELB DNS has been modified. Consequently, we've observed occurrences of error 1016 (CF DNS origin error). We are actively addressing this issue and appreciate your patience as we resolve it.

Our infrastructure provider blocked the problematic IP from being associated with our machines, and the issues related to this query ceased.

VTEX's team responds, confirming that only one IP was blocked and had been since 2021. However, this specific IP had started generating traffic volume from March 6th, coinciding with the start of the incident.

VTEX informs us of the blocked IP and indicates that action could be taken on Monday.

We contacted our infrastructure provider requesting the rotation of this IP and its exclusion from allocation to our sites.

We are continuing to work on a fix for this incident.First contact made with VTEX, accompanied by all collected evidence.

Throughout the day, manual interventions were carried out on the stores to reduce the number of errors, such as redeployments, blocking potential bots, and reducing calls to VTEX. Efforts were also made to enhance error logging to pinpoint the root cause.

The team observes a recurrence of the error across different stores, with the common denominator being the integration with VTEX. Initial suspicions were towards VTEX throttling, as not all VTEX customers were affected, and those who were did not experience it 100% of the time.