Additionally to this permanent weight penalty that applies whether or not four wheel drive is engaged, actually engaging all four wheels adds yet another fuel penalty. The engine will have to work harder to overcome the extra friction and resistance in the drivetrain introduced by the engagement. Additionally, the extra load on the engine will induce larger losses internal to the engine, such as crankcase losses.
If you need four wheel drive in your working vehicle, go right ahead and get it. If you need it, you need it. If you are looking for a recreational vehicle that you will use for playing off road, go right ahead. But if you are thinking about buying a four wheel drive vehicle without really knowing when you would use it just because it sounds like it would be "nice to have", remember that you will be paying extra at the pump for the rest of your ownership. It may be "nice to have", but so would be the gas and money you could have saved.