I had a little time tonight the other night to look at the carb some more, I'm still having the problem of not wanting to idle once its warmed up. A few people have told me to look at the gasket between the carb and intake.
The gasket was sticking out the side, and didn't look real good so I thought that it would be a perfect place to start. When I took the carb off this is what I found...

Nice! They are rubber and look like they where cut out of a bike intertube using a butter knife. I made a new gasket using actual gasket material, but I still get the same results, won't idle when its warmed up enough for the choke to be off.
I think that I'm done with the carb for a little while, I need to start working on the body so that I can get it painted this year.