I would say to keep persevering in saying it together with your husband. Of course the devil will try whatever means he can to stop you saying it together.
I don't know what time you said it before, but my husband and I always said it after breakfast and right before bed. I always just planned on nursing at that time. If baby was asleep, we'd start; then if baby woke up I'd get it and do a diaper change, then come back to the room and nurse. (I always covered with a blanket for modesty). If baby got very fussy, I'd go in the next room and sit (where I could still hear) or in the bedroom and finished up there.
When our 2nd came, my husband took over the 1st baby, and made him sit still in front of him during prayers. Every time we have a new baby, that's the scenario.
This way baby learns from the beginning about being quiet during prayers, and you set a solid habit of the FAMILY Rosary.
PS: Don't beat yourself up about falling asleep during the Rosary, either. I do it often, to my embarrassment. It's literally the only time I get to have some quiet in the house, and the prayers are so monotonously peaceful...St. Therese the Little Flower often fell asleep during choir, I found out recently. Just offer it up and try harder the next time to stay awake. Maybe if you're holding the baby, you can quietly walk back and forth behind your husband, so that you don't distract him. I've done that too, with fussy babies.
Good luck!