What is a good fabric to sew with fleece?

I'm making baby blankets and using a cute cotton pattern on the other side of the fleece. It just seems like when I sew it, the two fabrics don't sew well together and what I thought was a nice clean straight stich becomes crooked. It's like the two fabrics got thru the machine differently. IDK. I can usually sew a straight line. Anyways, is this a common problem or just me? Also would flannel sew better to fleece?