I am experiencing this as well. When I watch an HD mkv file with VLC I see jaggies as if it isn't AA but when I take a snapshot with vlc the picture is perfectly smooth. If I watch my videos with SMPlayer there are no jaggies either.
I have an ATI HD 3850 1Gb card.