How to get all the lint off of a fleece sweater?

Every time I wear my uniform pants with my white fleece jacket, the white lint attaches to them and it's pretty unsightly? Any idea so that it won't ever happen again?