Ideally, pots--especially porous ceramic and terra cotta--should be emptied and cleaned in the fall and stored in a protected area like a shed or garage. Moisture in the pores will freeze, weakening pots and causing them to chip and flake.
Although it can be inconvenient and time consuming, it's a good idea to remove all soil from pots and window boxes and replace it with fresh soil every spring, rather than just pulling out the dead plants and putting new plants in the old soil. Old soil is probably compacted, losing a lot of the necessary pore spaces for air, and may have accumulated toxic salts from last season's fertilizing and watering. Leftover soil may also contain disease-causing organisms, weed seeds or insect eggs.
Clean and scrub pots with a firm-bristled brush and rinse with a strong spray of water from a hose. Sanitize smaller pots by brushing and rinsing out any soil and washing them in the dishwasher without detergent. Thorough cleaning reduces the spread of pests and plant diseases.