No offense, but growing at night is IMPORTANT! Eat, rest, grow! Subtract an ounce of meat here and there from earlier in the day and tack it onto the end. It's doable.
A lot of people drink a shake after they lift, then an hour later have a meal, because they've already used it up! What you're doing is using it all up and starving your body all night, but your body needs that to build muscle. More muscle, more metabolism, more fat burning.