"Our situations differ." A warden would, reasonably, have a very good reason to lock a door. They likely had access to all KINDS of things that inmates generally did not; weapons, magic items, the possibilities were endless.

For a warden, not using a lock was probably the height of irresponsibility. "I... can't say I have anything important to me here, besides people. And I can't very well lock them up in my room." His tone turns that last part into a joke, he's certainly not even THINKING about it. "I suppose it's for the best. But I'll endeavor to be there when you turn up, either way."