From my point of view I'm not aware if there's a rule with the sound, you just have to know the language well to understand it, but a general rule if you do know Arabic is if the word sounds the same in Arabic but it's a K sound there, then you know it's a Kap, but if the Arabic sound is "Khet" or "Het" then you know it's a Khet, not a Kap.
A few examples:
Haleeb - Khalwa - Milk
RooH - Rookha - Soul
Hayat - Khayoota - Life
AlayKum - Alokhun - Upon you
MalaaK - Malaakha - Angel
MubaraK - Mburkha - Blessed