'03 5.4 idle acting weird???
 

180k miles.

The idle has slowed down considerably, both at start up and normal idle. Also when stopped at red lights the idle sometimes goes up and down about 200 rpm for a few seconds. Everything else is acting ok, no acceleration issues.

Think it's the fuel pump or filter?

Possibly the filter, more likely a dirty or sticky IAC valve.