Not only would I swim in it, I HAVE swum in it. www8bridges.org. 8 Bridges swum in 7 days. 120 miles, ending up at the Verrazano. The swim around Manhattan?
as a sailing instructor and avid boater, I spend many days during the summer months in the river. I find it to be clean and as long as you are safe ( pfd's, not going alone, the ability to self rescue or get help if you need it, an understanding of the strong tidal currents that run in the river, and not be in somewhere you shouldn't) the river is fine to swim in....I would not worry about the river being "polluted" or "unsafe"
There are definitely places on the river that are safe to swim, one of them is River Pool at Beacon! It's tested weekly by the Depatment of Health and is a great place to get cooled off and have some fun in the river, there is a lifeguard on duty too!