People are often surprised when they see me holding my graduated neutral density filter in my hand instead of putting it in a holder. The reason is simple: speed. When conditions are changing rapidly, or even when they aren’t, it’s a lot faster to hold the filter in front of the lens. If I change lenses, I don’t have to take the time to remove holder from one lens and place it on the other. I can reframe or switch from horizontal to vertical in an instant.
You need to watch out for reflections, though. The filter will reflect light if pulled away from the lens too far.