StorM - Missing EANs causing upload issues – Incident details

Search experiencing partial outage

Missing EANs causing upload issues

Resolved
Operational
Started 11 months agoLasted 1 day

Affected

Job Service

Operational from 8:29 AM to 7:27 PM

Upload Handler

Operational from 8:29 AM to 7:27 PM

Product Data Importing

Operational from 8:29 AM to 7:27 PM

Updates
  • Resolved
    Resolved

    StorM has now caught up on the backlog of EAN updates.

    Short story time:

    At around 15:30 (CEST) we hit the EAN updates which StorM had already seen before, which resulted in StorM catching up on processing now.

    Since Wednesday there have been around 500,000 new EANs being created in StorM. These may not necessarily be new to BESTSELLER, but is the first time they were sent to StorM from another system.

    From our observations, StorM began to receive a big load of EANs on Wednesday 1st Nov, at 19:00. By the following day, Thursday at 15:00 StorM was sent a total of 7 million EAN updates. While this was certainly out of the ordinary, we thought that over the weekend there would be enough time to catch up. However, in the morning Friday at 09:00 and up until 21:00 StorM received a further 5 million events. This meant the backlog of updates of EANs was close to 12 million.

    We'll be continuing to investigate, in order to counter similar events, should they occur.

    Again, we're sorry for the inconvenience.

  • Monitoring
    Monitoring

    We would like to provide you with an update on the issue some of you are facing regarding the uploading of images for specific EANs. We understand that this issue has been causing inconvenience to you and we are truly sorry for that.

    We have since made some significant changes in our processing system which will allow us to handle up to four times the normal amount of EANs. This has definitely sped up the processing of EANs. However if an EAN is new to StorM, it takes longer to handle than EANs that have been seen by StorM before. Which makes it difficult to predict how much longer it will take to catch up completely.

    Typically, we can handle up to 100,000 new EANs a day, which is more than suitable. However we've seen in the order of millions of EANs being sent to StorM on Friday, and over the weekend. We were not made aware of any such updates coming from other systems, and as such could not prepare for the amount of data in time.

    If we are to assume all updates that are stuck in the processing queue are new EANs, then we'd calculate that it will take some days. However this is unlikely the case, as it's not typical that such a large amount of EANs are created in normal circumstances. Due to this, it's unfortunately very difficult for us to provide an accurate ETA.

    We understand that this can be frustrating for you and we want to thank you for your patience and understanding through this difficult time. Our team is working round the clock to resolve this issue, and we will continue to monitor the situation closely.

    Please be assured that we are doing everything in our power to fix this issue as quickly as possible. Thank you for your continued support and please feel free to reach out to us if you have any further issues that we can help with.

  • Identified
    Identified

    We are aware that some users are having issues uploading images for specific EANs. The issue is due to a sudden large amount of EANs being sent from other systems, causing a backlog on the import into StorM.

    We are monitoring and working to improve the performance of the import systems. We will update here when progress is made towards resolving this issue.