Emil Sayegh is a serial Tech CEO with over 12 years of experience as a Private Equity backed CEO and more than 25 years in the IT and tech industry.
This incident raises critical questions about how we should react and prevent such occurrences in the future, whether it involves CrowdStrike or another entity.Let's start with what we know from CrowdStrike's somewhat opaque statements.
. CrowdStrike, a US cybersecurity company based in Austin, Texas, provides ransomware, malware, and internet security products primarily to businesses and large organizations. On Friday, July 19, at 4:09 am UTC , CrowdStrike released a sensor configuration update for their Falcon software program, targeting Microsoft Windows systems. According to a statement on the company's blog, the Falcon sensor is a cybersecurity software agent that offers automated protection against malware, antivirus support, incident response, and other security features. Falcon operates as a cloud-based program that works in conjunction with CrowdStrike's servers without requiring customers to install and manage additional local software on PCs and servers. CrowdStrike noted that updates to the Falcon program occur multiple times a day and have done so since the program's inception.NYT ‘Strands’ Hints, Spangram And Answers For Tuesday, July 30thThe update, designed to target malicious system communication tools in cyberattacks, triggered a"logic error" that resulted in an operating system crash on Windows systems . A logic error means a mistake in coding caused a bug, leading to not only a program malfunction but also a total crash of the host Windows machines.to improve the agent rather than cause issues. However, the patch did not go as planned, causing millions of Windows PC users to encounter the"Blue Screen of Death" on their devices, with many computers entering a non-stop reboot loop. CrowdStrike confirmed that anyone using Falcon on Windows version 7.11 or above was affected. As a consequence of this incident, CrowdStrike shares closed trading at $256.16 on Friday, July 26, down significantly from $343 on July 18 before this incident.The problematic file, known as a channel file, needed to be deleted. If the systems were online, it could be deleted remotely. If offline, IT support had to intervene manually. Not all users were able to delete the buggy file remotely, necessitating manual deletion for the device to function again. Channel File 291 was the impacted file, causing disruptions across various sectors, including grounding commercial airline flights, temporarily taking Sky News and other broadcasters offline, and disrupting banking, healthcare services, and 911 emergency call centers. Although CrowdStrike issued a patch to fix the error, computers stuck in a boot loop were unable to connect to the internet to download the patch. The recommended solution from CrowdStrike was to boot into safe mode or Windows Recovery Mode and manually delete Channel File 291, requiring local administrator access and a BitLocker recovery key if the device was encrypted.Why was the patch not tested on a small number of PCs before being widely deployed?Why was CrowdStrike’s response so dismissive and inadequate?Many DevOps professionals live by Facebook founder Mark Zuckerberg’s now-famous motto: “Move fast and break things.” This mentality, intended for internal design and management processes, has also shaped how DevOps professionals view disruption: failing faster is always better. However, this approach is increasingly untenable when it comes to systems that impact life-saving services such as hospitals. Many patients were unable to have surgeries that day. While this may work for Facebook or a gaming platform, it does not work for industries where lives are at stake.In the technology sector, boards must drive this change if CEOs cannot, are too vested, or are unwilling to. The AI technologies of tomorrow—military drones, medical robots, and more—will impact lives far more than those of the past ten years.The recent CrowdStrike incident is a wake-up call. As we rush to innovate, we must not overlook the essentials of quality control and accountability. Only then can we prevent such chaos and ensure the reliability and trustworthiness of our technology systems. Technology is far-reaching and touches medical, transportation, and military fields where lives are in the balance.Quality Assurance and Quality Control , while unglamorous, are foundational practices that must be rigorously upheld. It’s crucial to have multiple layers of testing, including automated tests, manual reviews, and beta testing on a limited number of systems before a full-scale rollout. This helps catch errors before they escalate into widespread issues, even with emergency security patches. The medicine cannot be worse than the cure. Organizations must adopt a contrite and introspective approach when responding to mass outage events. Instead of downplaying the issue, as CrowdStrike initially did, they should acknowledge the problem, provide transparent explanations, and outline steps being taken to rectify and prevent future occurrences. This builds trust with customers and stakeholders. There must be accountability within organizations for errors that cause significant disruptions. This includes not only technical staff but also management teams. Clear protocols should be in place to address mistakes, ensuring that those responsible understand the impact of their actions and work towards preventing future incidents. Has anyone been disciplined? Have processes been revised and revisited? The offering of $10 gift cards by CrowdStrike as compensation for significant service disruptions is insufficient and dismissive. Companies must provide proper adequate monetary compensation that reflects the inconvenience and potential losses suffered by their clients. This demonstrates genuine concern and responsibility for the incident.Continuous training is vital for both technical and non-technical staff. Regular training sessions on best practices, new technologies, and incident response protocols can help prevent mistakes. Additionally, simulations of potential issues can prepare teams to handle real-world problems more effectively.The CrowdStrike incident is a stark reminder that in our quest for innovation, we must not lose sight of the fundamentals of quality control and accountability. Pushing software updates is not to be relegated to amateurs or beginners. We need to return to these original principles that have made our industry dependable. Bringing in experts who specialize in outage management, outage communication, and root cause analysis is essential. These professionals, should answer to the board directly, and provide valuable insights and strategies for managing and mitigating the impact of outages, ensuring a more resilient and responsive system. Boards must mandate these changes if C-level executives are unwilling to do so. Just as CrowdStrike is brought in as the gold standard for cybersecurity breaches, the same rigor should apply to handling internal and external outages. This incident is a wake-up call: to prevent chaos and maintain the integrity and trustworthiness of our technology systems, we must prioritize quality control and accountability, especially in sectors where lives are at stake.Our community is about connecting people through open and thoughtful conversations. We want our readers to share their views and exchange ideas and facts in a safe space.Insults, profanity, incoherent, obscene or inflammatory language or threats of any kindContinuous attempts to re-post comments that have been previously moderated/rejectedAttempts or tactics that put the site security at riskProtect your community.
Cybersecurity Software Update Falcon Sensor Logic Error Blue Screen Of Death IT Systems Crash Quality Control Devops Incident Response
United States Latest News, United States Headlines
Similar News:You can also read news stories similar to this one that we have collected from other news sources.
Republican representatives call on CrowdStrike CEO George Kurtz to testifyRepublican representatives called on CrowdStrike CEO George Kurtz to testify about its defective software update.
Read more »
CrowdStrike CEO to testify about massive outage that halted flights and hospitalsThe House Homeland Security Committee called on CrowdStrike CEO George Kurtz to testify on what happened in the massive outage that impacted several sectors.
Read more »
CrowdStrike CEO called to testify to Congress over cybersecurity's firm role in global tech outageA defective software update sent by CrowdStrike to its customers disrupted airlines, banks, hospitals and other critical services.
Read more »
CrowdStrike CEO called to testify to Congress about global IT outageThe Austin company said Monday it was testing a new technique to accelerate remediation, but declined to say whether CEO George Kurtz would testify.
Read more »
CrowdStrike CEO says 97% of Windows sensors back online after major outageLast week, CrowdStrike issued a routine update to its users around the world that contained a bug which caused Microsoft’s Windows operating system to…
Read more »
CrowdStrike CEO says 97% of Windows sensors are back onlineCrowdStrike CEO George Kurtz posted on LinkedIn that the company is still working to ensure all systems have been fully recovered after an issue led to a global tech outage.
Read more »