Stress is a major consequence of endeavor. If you want to make something happen, naturally you're going to worry about it until its done.
Successful people worry about performing consistently all the time. That's why they practice every day, constantly looking for ways to strengthen the fundamentals and allow for creative flexibility.
Anxiety also holds true for listlessness. While lazy people avoid stress in the short-term, they're regrets pile up in the long-term. They die with sadness, leaving little significant impact on the world.
Stress is prevalent in both doing the work and idleness. Either get active and take chances or suffer through regret.
The misery is in your own making. Good luck.