Duration: 1 day, 14 hours and 58 minutes

Affected Components:

Aug 4, 12:42:35 GMT+0
Resolved - This incident has been resolved, and all valid data has been processed. If you find any issues with missing data, please reach out to support..

Aug 4, 06:36:31 GMT+0
Monitoring - The system has processed all outstanding data and is back to normal. Note that a few uploads and copy operations have failed, and we will try to resolve and rerun them to resolve this issue fully..

Aug 3, 10:40:13 GMT+0
Monitoring - We implemented a fix and are currently monitoring the system. Please note there's quite a lot of updates in queue, which will may take up to 24 hours to process. .

Aug 2, 21:45:00 GMT+0
Investigating - We are aware of an issue in updating data in StorM, and we are investigating. Rest assured that any uploads and data modifications you may make is being saved in the system, and will be displayed once this issue is resolved.

Duration: 41 minutes

Affected Components:

Jan 17, 09:12:49 GMT+0
Resolved - A fix has now been deployed. We're sorry for the inconvenience caused..

Jan 17, 08:31:57 GMT+0
Identified - You may encounter some issues validating EANs when attempting to upload to StorM. We have already identified the issue, and are working on a fix..

Duration: 2 hours

Affected Components:

Dec 17, 08:00:00 GMT+0
Completed - Maintenance has completed successfully.

Dec 17, 06:00:01 GMT+0
Identified - Maintenance is now in progress.

Dec 17, 06:00:00 GMT+0
Identified - We need to perform important security updates to StorM. During this time, our services will unfortunately need to be made temporarily unavailable for up to 2 hours. In order to implement these security improvements, we need to rebuild part of our core infrastructure from the ground up. While we always try to avoid any service interruptions, this particular upgrade requires us to temporarily take the system entirely offline to properly implement these security enhancements. We know it's never convenient when our service is unavailable, and we're sorry for any disruption this may cause. We've scheduled this work during quieter hours to affect as few of you as possible, while giving enough working time to resolve any potential issues after the upgrade. If you have any questions or concerns, please reach out to us [on Teams](https://teams.microsoft.com/l/team/19%3Ab222364bda8d4d378f7d0a95a8d15663%40thread.tacv2/conversations?groupId=6c85b0c3-d9c9-47ed-9080-9fcff41dba3f&tenantId=98db9fb9-f52b-4e63-83d9-795ccd2dfcca) or to us [via support](https://storm.bestseller.com/support)..

Duration: 2 hours and 29 minutes

Affected Components:

Dec 9, 09:15:25 GMT+0
Resolved - StorM is continuing to operating normally. The backlog of pending files is being processed as expected. All systems appear stable and no further disruptions are anticipated..

Dec 9, 08:25:58 GMT+0
Monitoring - Good news! We've resolved the technical issues, and StorM is now working properly again. It's currently processing the pending files that built up during the downtime. We'll keep watching the system until all delayed files are processed and everything returns to normal operation..

Dec 9, 07:47:01 GMT+0
Identified - We have identified a couple of issues and are in the process of applying fixes..

Dec 9, 06:46:37 GMT+0
Investigating - We are currently experiencing issues with our data processing pipeline in StorM. This is affecting both the distribution of content to connected systems and updates to the StorM search. We are identifying the root cause related to our message processing system and is actively working on a resolution. During this time, you may notice delays in content synchronization and search results not reflecting recent changes. We apologize for any inconvenience and will provide updates as soon as we have more information..

Affected Components:

Sep 10, 07:32:20 GMT+0
Resolved - A mismatch between user permissions and logic applied in StorM Search resulted in showing no results. The change has been reverted for now, while a long term fix is looked into..

Duration: 57 minutes

Affected Components:

Jul 29, 06:27:11 GMT+0
Completed - Upgrade completed, thanks for your patience 😊.

Jul 29, 05:30:01 GMT+0
Identified - Maintenance is now in progress.

Jul 29, 05:30:00 GMT+0
Identified - We're performing essential maintenance to improve our system's security, and performance. This upgrade will result in approximately 1 hour of downtime. We appreciate your patience as we work to enhance your experience 🙏.

Jul 26, 07:11:00 GMT+0
Identified - Our apologies, correction to the time; 07:30 AM..

Duration: 30 minutes

Affected Components:

Apr 29, 06:30:00 GMT+0
Completed - Maintenance has completed successfully.

Apr 29, 06:00:01 GMT+0
Identified - Maintenance is now in progress.

Apr 29, 06:00:00 GMT+0
Identified - We would like to inform you of an upcoming scheduled maintenance window. To ensure the best performance and security of our systems, we will be performing updates on the servers that StorM runs on. Details of the maintenance window: \- Start time: [Monday 29th April 2024, 08:00 (CEST)](https://www.timeanddate.com/worldclock/converter.html?iso=20240429T060000&p1=1405&p2=136&p3=155&p4=tz%5Faet&p5=179) \- Expected duration: Approximately 30 minutes or less During this maintenance period, you may experience temporary disruptions or inability to access to StorM. We expect the maintenance to be completed within the estimated 30-minute window. However, please note that unforeseen issues may extend the maintenance period. We will do our best to minimize any extended downtime. If you have any critical tasks or processes dependent on StorM, we recommend planning accordingly and scheduling them outside of the maintenance window. Thank you for your understanding and continued support. If you have any questions or concerns, please don't hesitate to reach out to our support team at <https://storm.bestseller.com/support>..

Duration: 19 days, 17 hours and 59 minutes

Affected Components:

Feb 19, 11:14:57 GMT+0
Resolved - We have mitigated the ongoing issue by changing some configuration with our current provider. Meaning that content from now should be delivered in the quality you expect for the "original" content types. While the initial issue has been resolved, there are some consequences that we will continue to address; 1) Determine a point in time that we believe the issue came about (Work in progress) 2) Based on the point in time, attempt to re-deliver content to other systems that are affected during this period of time 3) Continue talks on alternative options from our current provider 4) Identify ways in which we can monitor that original files are being delivered by the provider in original formats as expected If there are any questions or continued support required relating to this, then you're welcome to use the "StorM" Team in Microsoft Teams to discuss. Otherwise please reach out to support via <https://storm.bestseller.com/support>..

Feb 16, 08:56:37 GMT+0
Identified - We still have not received any further updates relating to an ETA or fix relating to the ongoing issue at our current provider. Due to the length of time this unresolved issue has now taken, we have began to have talks with another provider which will continue early next week. Our initial testing indicates that this new provider could meet some of the specific requirements that StorM needs. As such, we're hopeful that our talks next week will end well..

Feb 13, 12:37:39 GMT+0
Identified - We reached out for another update, in which we received: "The issue with the edge rule is currently being reviewed and addressed by our dedicated team. Rest assured, we will update you with any new developments directly through this ticket." We'll continue to request for updates. Meanwhile we're considering alternative platforms going forward, as this level of service/communication is unacceptable..

Feb 9, 09:59:42 GMT+0
Identified - A response was received from our provider: "Apologies for the inconvenience. I understand the importance of your request however, we unfortunately do not have an update at the moment from our development team yet. While we do not have an ETA when this will be resolved, we'll notify you as soon as possible when we have more information to share." We'll continue to try escalate the problem where we can..

Feb 9, 08:16:01 GMT+0
Identified - We have not received any further communication from our provider. As such, we have reached out again for an update..

Feb 5, 07:45:00 GMT+0
Identified - We reached out again to our provider, as we have had no update regarding any news/updates from the developers there. We received the following update: "I am afraid that we currently do not have a solution for this and our development team is still working on a fix. We will make sure to let you know as soon as possible.".

Feb 1, 12:07:44 GMT+0
Identified - We have received a not so great response still, lacking any further information: "Unfortunately we'd not be sure of an exact timeframe, my apologies about that. I will however chase this with our developers for you." We'll wait for any further update in relation an update from their developers..

Feb 1, 11:31:43 GMT+0
Identified - We have reached out to the provider for another update..

Jan 31, 11:26:23 GMT+0
Identified - A response to asking for an ETA was just received, as quoted: "Unfortunately, we still don't have an ETA when this will be resolved. Apologies for that." We'll continue to wait for any further updates today, or otherwise reach out directly again tomorrow. If you need to download content from StorM, then we'd advise you to download via the Basket. The Basket downloading does not rely on this 3rd party provider, and as such delivers files to you in the original upload quality..

Jan 31, 10:41:31 GMT+0
Identified - We have reached out to the 3rd party provider for an update on the situation..

Jan 30, 17:16:00 GMT+0
Identified - We were notified of a discrepancy between images uploaded to StorM, and what was available to download (via the "One Click" download option), as well as delivered to other systems. After analyzing the problem, it appears a 3rd party provider that is used to deliver StorM's content (Content Delivery Network) is performing optimizations on images, even though StorM has an explicit rule defined for these optimizations to not be applied. An issue was raised with the provider, and they have since confirmed and identified the issue. However no ETA was provided for when this will be fixed. We'll update as we learn more about the siutation. .

Affected Components:

Nov 30, 09:16:57 GMT+0
Resolved - Search is back up and running, sorry for the inconvenience. Some unexpected issues happened when applying an update, which is aimed at reducing downtime of search in the future. Going forward, we now have two additional servers providing search functionality. This should mean that downtime in the future would be to an absolute minimum (or no downtime) when it comes to making future changes to search, or updates to StorM servers. If that makes no sense, then here's a visual representation. You can imagine search being the finger, it just keeps coming back: .

Duration: 3 hours and 11 minutes

Affected Components:

Nov 29, 18:25:51 GMT+0
Resolved - The database has been restored to full operational state.

Nov 29, 18:16:24 GMT+0
Monitoring - The database is restoring to its complete state, but is operational once again..

Nov 29, 15:15:00 GMT+0
Identified - After upgrading the servers, the database failed to start up correctly..

Duration: 28 days, 10 hours and 21 minutes

Affected Components:

Nov 8, 19:27:30 GMT+0
Resolved - StorM has now caught up on the backlog of EAN updates. **Short story time:**  At around 15:30 (CEST) we hit the EAN updates which StorM had already seen before, which resulted in StorM catching up on processing now. Since Wednesday there have been around 500,000 new EANs being created in StorM. These may not necessarily be new to BESTSELLER, but is the first time they were sent to StorM from another system. From our observations, StorM began to receive a big load of EANs on Wednesday 1st Nov, at 19:00. By the following day, Thursday at 15:00 StorM was sent a total of 7 million EAN updates. While this was certainly out of the ordinary, we thought that over the weekend there would be enough time to catch up. However, in the morning Friday at 09:00 and up until 21:00 StorM received a further 5 million events. This meant the backlog of updates of EANs was close to 12 million. We'll be continuing to investigate, in order to counter similar events, should they occur. Again, we're sorry for the inconvenience..

Nov 8, 08:48:37 GMT+0
Monitoring - We would like to provide you with an update on the issue some of you are facing regarding the uploading of images for specific EANs. We understand that this issue has been causing inconvenience to you and we are truly sorry for that. We have since made some significant changes in our processing system which will allow us to handle up to four times the normal amount of EANs. This has definitely sped up the processing of EANs. However if an EAN is new to StorM, it takes longer to handle than EANs that have been seen by StorM before. Which makes it difficult to predict how much longer it will take to catch up completely. Typically, we can handle up to 100,000 new EANs a day, which is more than suitable. However we've seen in the order of millions of EANs being sent to StorM on Friday, and over the weekend. We were not made aware of any such updates coming from other systems, and as such could not prepare for the amount of data in time. If we are to assume all updates that are stuck in the processing queue are new EANs, then we'd calculate that it will take some days. However this is unlikely the case, as it's not typical that such a large amount of EANs are created in normal circumstances. Due to this, it's unfortunately very difficult for us to provide an accurate ETA. We understand that this can be frustrating for you and we want to thank you for your patience and understanding through this difficult time. Our team is working round the clock to resolve this issue, and we will continue to monitor the situation closely. Please be assured that we are doing everything in our power to fix this issue as quickly as possible. Thank you for your continued support and please feel free to reach out to us if you have any further issues that we can help with. .

Nov 7, 08:29:10 GMT+0
Identified - We are aware that some users are having issues uploading images for specific EANs. The issue is due to a sudden large amount of EANs being sent from other systems, causing a backlog on the import into StorM. We are monitoring and working to improve the performance of the import systems. We will update here when progress is made towards resolving this issue..

Duration: 30 minutes

Affected Components:

Oct 23, 08:04:20 GMT+0
Resolved - This incident has been resolved..

Oct 23, 07:34:22 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Duration: 2 hours and 52 minutes

Affected Components:

Aug 19, 15:51:54 GMT+0
Completed - Search has now completely been re-populated with data, and is updated to latest version. .

Aug 19, 13:41:42 GMT+0
Identified - Search has been updated, and is partially operational. The data is being re-built for which will take a few hours, but should complete within the maintenance window..

Aug 19, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Aug 19, 13:00:00 GMT+0
Identified - We're going to update the search in StorM to the latest available version which brings some performance improvements, and additional features we can potentially leverage in StorM. Switching to the new version will not take long, however we need to re-build the search data which will take a few hours, meaning **only your search experience will be affected**. This does not impact any other flows within the system. After this update, we should be able to make such upgrades without introducing downtime (*a new feature*). .

Affected Components:

Aug 18, 11:36:18 GMT+0
Resolved - A misconfiguration in the Load Balancer in preparation for update to search hit Production. This has been reverted..

Duration: 2 hours and 45 minutes

Jun 6, 09:34:33 GMT+0
Resolved - Still no update from Networking Team, however we will mark this as resolved after observing uptime for an hour now..

Jun 6, 09:07:22 GMT+0
Monitoring - We're observing normal access to BSDAM. No update from Networking Team has been provided yet. If we continue to see access as normal, we will mark this as resolved..

Jun 6, 08:32:17 GMT+0
Identified - We're observing intermittent access to BSDAM, however no update from the Networking Team has been provided yet..

Jun 6, 06:50:00 GMT+0
Identified - The Networking Team are investigating DMZ Firewall issues which affect the Load Balancer which provides access to BSDAM. We need to wait for Networking Team to resolve the issues. The "Hot Folders" and distribution of images should remain unaffected. However new product data (for example new EANs) may be delayed affecting new uploads..