Sleep doesn't only affect the amount of energy you'll have the next day, but it also affects the regulation of the hormones of the body. Not getting enough sleep consistently causes a huge hormonal imbalance and that can contribute to your testosterone levels dropping drastically.