The parents expressed their care for their children in the avenues they preferred to do so.

He doesn't need to make any assumptions of other people

Assuming you tried to avoid a double negative and instead meant "There's nothing wrong with them making assumptions of other people": Again, I'm not blaming him for making those assumptions. But he did believe it would not cause harm which most likely turned out to be wrong. And that was, in your narrative that I agree with, based on the reasonable assumption that there's no ulterior connotation to "Lolita".

You can argue that Japanese Lolita fashion has little do with the infamous book, but that doesn't shake the fact that we are not in East Asia but in North America, where "Lolita" is intensely associated with the book and pedophilia. The parents made a reasonable assumption that this is what "Lolita" meant. There's no difference here between the amount of assumptions made by the driver and the parents.

They can look away

From the reasonable conclusion that their driver is a predator?

unfairly forced this man out of his position

That's an assumption. It's still possible the driver is given a second chance at bus driving. And in the worst case I doubt the driver would not be able to find employment in public transportation.

The driver caused zero harm.

That's also an assumption. I don't see any reason why the driver would not cause parents to not trust the bus and harm the children forced either to wake early and walk to school or contribute to the emissions in their air.

or chose to drive their own kid to school.

Again, there's the harm. The pollution doubling as half the bus goes to school in a car instead and being prevented from talking with friends on the bus, especially those with whom one does not share a lunch period. And your assuming that all the parents have the time to drive their children to school without conflict from their work commute.