I decided on a belt-and-suspenders approach to sealing the gaps around my windows. First I put in backer rod, which is foam tubing that comes in different widths, then I used Great Stuff, a foam spray sealant.