Microsoft has been too busy building a new Outlook PWA with ads in your email, and AI laptops that capture screenshots of your desktop in unencrypted folders.
Windows ADK does this too, or any PXE server really... so yes, you can. The CS issue though didn't require re-image. Merely removing a file. DR planning would usually have a recovery image pre-installed to automate booting into for lower-level fixes.
Imaging environment down? If a sysadmin can't figure out how to boot a machine into recovery to remove the bad update file then they have bigger problems. The fix in this instance wasn't even re-imaging machines. It was merely removing a file. Ideal DR scenario would have a recovery image already on the system that can be booted into remotely, so there is minimal strain on the network. Furthermore, we don't live in dial-up age anymore.
Configure PXE to reboot into recovery image, push out command to remove bad file. Reboot. Done. Workstation laptops usually have remote management already.
or
Have recovery image already installed. Have user reboot & push key to boot into recovery. Push out fix. Done.
You were all in, but was the company all in? How many employees? It sounds like you innovated. Let's say that the company you worked for was spending millions on vendors that promised solutions but rarely delivered. If instead they gave you $400k a year, a $1 million/year budget & 10 employees.. I'm guessing you could have managed the laptop deployment automation, along with some other significant projects as well.
Instead though, people with good ideas, even loyal to the company, are competing against sales and marketing reps from billion dollar companies, and upper management are easily swooned.
Almost all computers can be set to PXE boot, but work laptops usually even have more advanced remote management capabilities. You ask the employee to reboot the laptop and presto!
Thank you for sharing this. This is what I'm talking about. Larger companies not utilizing something like this already are dysfunctional. There are no excuses for why it would take them days, weeks or longer.
I'd issue IPMI or remote management commands to reboot the machines. Then I'd boot into either a Linux recovery environment (yes, Linux can unlock BitLocker-encrypted drives) or a WinPE (or Windows RE) and unlock the drives, preferably already loaded on the drives, but could have them PXE boot - just giving ideas here, but ideal DR scenario would have an environment ready to load & PXE would cause delays.
I'd either push a command or script that would then remove the update file that caused the issue & then reboots. Having planned for a scenario like this already, total time to fix would be less than 2 hours.
I upvoted because you actually posted technical discussion and details that are accurate. PXE and remote power management is the way. Most workstation BIOS will have IPMI functionality already included. I agree thought that being that these are remote endpoints, it can be more challenging. Having a script to reboot their endpoints into a recovery environment though would be a basic step though in any DR scenario. Mounting the OS partition to delete a file & reboot wouldn't be a significant endeavor, although one that they'd need to make sure they got right. Still though, it would be hard to mess up for anyone with intermediate computer skills... and you'd hope these companies at least have someone trained to do that rather quickly. They'd have to spend more time writing up a CR explaining all the steps, and then joining a conference call with like 100 people with babies crying in the background... and managers insisting they remain on the call while they write the script.
Except these were endpoint machines, not servers. Things grinded to a halt not because servers went down, but because the computers end users interacted with crashed and wouldn’t boot, kiosk and POS systems included.
Endpoint machines still have IPMI type of interfaces and PXE. When you manage thousands of machines, if you treat them all like a pet then you're doing it wrong.
Damn, I guess all the IT people running the systems that were affected aren’t fit for the job.
Is it going to take them several days to weeks to recover? Then they aren't fit for the job, or should consider another profession.
Can you show me the bloated budgets and where they are allocated on those mid to large size corporations?
All of them. The Form 10k fillings are available for public corporations. The ones claiming that they will be impacted for a while are the ones I'm concerned most about.
It’s weird how you’re all defensive and devolve to insults when people are just responding to your post.
I spent a career arguing with sales reps who had one goal in mind, and that was to make the biggest commission possible. I sound argumentative because those sales reps had every tool imaginable to show up out of no where.
It was to elaborate that there is a bigger issue here with corporate IT culture that is broken. The CrowdStrike incident merely exposes it, but CrowdStrike isn't the real problem. Remediation for an event like this, especially once the fix is known, should be 30 minutes... not weeks or months.
Booting a system or recovery image remotely over an IPMI or similar interface is not complicated or expensive. It is one of the most basic server management tasks. You acting like the concept is challenging seriously concerns me and I seriously wonder how anyone that thinks like that gets hired.
There are exceptions, granted. However, the IT budget at most mid to large-size corporations is extremely bloated. I don't think you can in good faith argue otherwise, unless you want to show me a budget that isn't. Do you have a real one that you can provide?
These companies don't even attract smart talent. They attract people that are complacent with doing nothing & collecting a paycheck. Smart people do not continue to work at these companies. The bureaucracy and management is soul-sucking. It took me a while to accept it too. I used to be optimistic thinking there is a logical explanation that can be fixed. Turns out they don't want to be fixed. They like to be broken. Like I said, it starts from the top down. A lot of the staff wouldn't even have a job if people actually tried to make things better.
With most corporations, especially Fortune 500s... audit their budgets. The problem doesn't start with IT. but with bad management from top down. This "cost center" you speak of is mostly what I'd expect to hear do-nothing middle-level managers tell their in-house employees when asking for a raise.
Microsoft has been too busy building a new Outlook PWA with ads in your email, and AI laptops that capture screenshots of your desktop in unencrypted folders.