Yes you should really. All manufacturers recommend it. If the taps are always left on – and therefore under pressure – you could get a leak, or even a flood from a leaking or split fill hose.
This sometimes happens over night when reduced demand causes increased water pressure. More rarely, a sticking fill valve in the washing machine or dishwasher can allow water to enter the machine even when unplugged.
Big floods from a split fill hose pipe are less common these days because the hoses aren’t made of soft rubber any more. But if it does happen to you the consequences can be serious.
If your washing machine is fitted in any place where the temperature can drop below freezing such as a garage then you definitely always turn off the taps. I once experienced the tap and hoses freezing up in my garage. When the ice thawed the water started to gush out and completely flooded all my garage.
So if you can get to the taps easily then you should try to get into the habit of turning them off. If you can’t (many washing machines are plumbed in with the taps behind the washing machine) then there’s not a lot you can do. If you are concerned about this, you will need to get a plumber to move the taps to an accessible place so you can turn them off. I would estimate that the majority of people don’t turn off their taps.
Some washing machines come with anti-flood Aqua stop fill hoses so if you have these fitted it should offer some protection from split fill hoses.
If water goes into the washing machine or dishwasher overnight, or when not in use it may be a sticking water valve or a plumbing fault – see this article Washing machine fills with water overnight or when switched off