not necessarily. bait worms, for one, probably don't even reach the stomach of a caught fish, since the fish are effectively dead as soon as they bite the hook. they certainly don't have time to swallow and digest a worm before they're reeled in and suffocate.
second, when and if fish eat worms, they will most likely be annelids (earthworms - these are used for bait) which would pass through the digestive system and give nutrition. flat- and roundworm parasites, on the other hand, live in the flesh or intestine, are not digested, and take nutrition from the fish. i haven't gone out of my way to taste a parasite-infected fish to compare it to a normal fish, but i would imagine they are smaller, weaker and maybe even taste bad. if you contracted a parasite (let's say a tapeworm) you would lose quite a bit of weight because they eat a lot of your digested food before you get the chance to use it as your own food source (they can grow obscenely long, i don't remember exactly how long but it's more than 10 feet)
when you eat a cheeseburger, it ends up as energy and waste, you don't find huge chunks of intact cheeseburger hanging out in your muscles and liver.