That could be the problem, as the display, even though it's dimly lit, can still use up battery. Set the time to the lowest setting possible. Maybe this could be the problem.
Also, have you tried doing a full discharge/recharge? By discharging the battery to 0% and recharge the battery to 100% a couple of times, you could ensure a longer-lasting battery life in your Sidekick.
I could be wrong about discharging/recharging but I hope I can be of any help.