when and how to use set_clock_latency in RC

I have some confusion about when and how to use set_clock_latency in RC. I have a clock which goes through the pad and some buffers and then hits the flop. And I have some inputs to and output from the flops.

 CLK  ---> PAD_CLK --->CLK_BUF/Q

 I define virtual clock "CLK_v" to time both I/O together with external delay and I define real clock "CLK" which roots at CLK_BUF/Q.

Should I use set_clock_latency to take care of the delay due to PAD_CLK and CLK_BUF? If so, should I use -source with -early/-late or just use -min/-max. My understanding is that -soruce is only used for external clock insertion delay. For internal pad/buf delay, it's basically the same as network delay, hence I need to use -min/-max. However, when I use -min/-max, RC only uses the max value to timing the I2O, i.e., only use one corener "i.e., max". This is very different when I use -source where RC used  "-early" value for I2O. Is this expected?

BTW, for delay value used in "early/late", I got it for the clock path using min/max lib seperately. 

Or should I forget all this set_clock_latency business and directly define 'CLK' rooting at the CLK input port?

Any suggetions?