How to remove handle of a kitchen faucet?

I have a single handle kitchen faucet which is leaking. I wanted to put a new washer to see if that stops it. But i am currently stuck at the first point itself which is to remove the handle. I know that there is a set screw(which can be removed using a allen key) but cannot see it at all. There is a hot/cold symbol (picture) which somehow i think needs to be taken out to see the set screw but i am unable to do so. Any ideas what i am missing here?