I noticed a few things you might consider. I do not know your specific circuit parameters (for example, if frf1 = 5 GHz, is frf2 5 GHz +/- 40 MHz?, what are inductor values?), so my apologies if these do not make sense to you.
1. Your beat frequency appears to be 40 MHz. However, your PSS simulation time span is 25 ns - which is only one period of the beat frequency. Therefore, the pss analysis which looks for a steady-state solution, includes the initial settling time of your circuit. With at least 1 coupling capacitor C2 of 10 pf, it appears there will be a period of time at the start of the simulation where the operating points of devices will not be at their steady-state operating points. To estimate the final steady-state solution most accurately, I think it worth including a period of time at the beginning of the simulation (before the pss simulation starts) to allow for these transient conditions to settle to close to their steady-state values. The parameter tstab shown on page 3 of your document contains no value and hence is set to 0. I might suggest running a transient analysis and examing how long it takes the nodes in your circuit to settle to a steady-state value and use this value for tstab. The fact that you are observing many overvoltage values in your output log further suggests your circuit has not settled to a steady-state solution before starting the pss solution effort.
2. The input frequencies appear to be at 5 GHz and (possibly) 5 GHz +/- 40 MHz. Hence the period difference of the two waveforms is 1/( 5 GHz) - (1/(4.96 GHz)) = 1.61 ps. Your simulation, therefore, needs to accurately estimate waveform periods whose difference is 1.61 ps or less. However, I noticed you have not set the maximum integration timestep of your simulation from its default value and you are using an "errpreset" of "moderate" (page 3). With a "moderate" value for errpreset, and the number of harmonics you ahve set, the value of maxstep is shown as 1 ns (page 5). I would suggest reducing the value of maxstep from 1 ns to something on the order of the difference in the periods of the two applied frequencies or less. If you decide to set tstab to a non-zero value, you will also want to make sure the value of maxstep is also set to the same value in your intiial transient simulation (prior to starting the pss analysis).
3. Finally, you may decide to also set errpreset to "conservative" in lieu of "moderate" to further improve simulator accuracy. With the time constants you appear to have in your circuit, this appears to be a set of stiff differential equations that will stress the solver.
I am sure others will have good suggestions, but I hope this helps a little.
Hi Shawn, thank you for paying attention on my problems and your suggestions. My frf1 is 5GHz and frf2 is 5.04GHz. I tried to run transient analysis but the simulation keep running nonstop and never come to the end. My inductors value are as follow:
May i know what are the possibilities that caused the PSS analysis unable to converge? I am a totally beginner in using Cadence hence my knowledge is very limited. Hopefully you can kindly share your knowledge and experience here. Thank you.( i have attached my transient analysis simulation log)
> I tried to run transient analysis but the simulation keep running nonstop and never come to the end.
Perhaps I was not clear in my responses  and  and I am sorry for that! From your simulation log, it appears you did not follow my suggestions as the value of maxstep is much larger than my suggested value of less than 1.6ps and the value of relotl = 1e-3 (=> errpreset is "moderate", not "conservative").
If you look at your transient simulation log in your most recent post, the value of maxstep is set to 200us with a value of tstop of 10 ms. If you look at the timesteps being chosen in the output log, they are on the order of 10 ps - which is 20 periods of your 5 GHz input clocks!!
If your transient analysis does not produce accurate results, the pss analysis will also not produce valid results.
As I noted in response , the difference in the period of your two 5 GHz and 5.04 GHz inputs is on the order of 1.6 ps. You need to set maxstep (the maximum integration step) in the transient panel under "options" to, for example, 1 ps or less and choose an errpreset on the Transient Analysis GUI of "moderate", or perhaps better "conservative". You also do not need to run the simulation for 10 ms - that is far too long to observe the time where a steady-state solution is evident.
I hope these suggestions make more sense now Xujing.
Shawn & Xujing,
I'm not convinced that these suggestions are necessary. You should not have to mess with maxstep for a driven circuit, and you should not need to give tstab to be where it has settled - at least for a circuit with "linear" settling. PSS (both shooting and harmonic balance) should solve for the steady state, and do not have to be at the steady state before they start. I have simulated cases where it would have taken 10,000 cycles to settle without needing a tstab. Generally tstab can help, but is mainly useful when the start up behaviour is not simple RC-type settling.
There are a few things that look odd. First of all, the picture of the psin form (BTW, I would generally recommend using the "port" component from analogLib rather than the rather outdated "psin" component) doesn't show the value for "Frequency 2". I suspect it is not set - which is why on the PSS form itself it only shows a single tone (at 5GHz). So only a single frequency is going into the LNA. If this had been set correctly, the auto-calculate would have shown the common frequency between the two inputs. Setting the common frequency to 40M won't help if you only have inputs at 5G because those sub-harmonics won't appear.
Assuming that you meant to have a second frequency at frf+40M, then 127 harmonics of 40M would only give you up to 5.08GHz. That's fine - because shooting PSS would give you spectral accuracy by default up to 4 times the largest harmonic requested (this is more for small-signal analyses that might follow, but the maxstep during the shooting interval would be set to something like 9.84ps (1/(PssFund*maxharm*4*5)). So you shouldn't have to do anything special - this should be fine - if smaller timesteps are needed to follow the waveforms, the simulator will do that anyway. You could set maxacfreq higher (say to 15G or something like that) if needed - but it shouldn't be needed. The maxstep that was reported in the PDF was actually from the tstab part, and that has looser timestep requirements (doesn't need to be so accurate).
Conservative is a good idea if you're trying to look at IM3 products (say), because these can be quite small and you need the resolution.
However, something is fishy - the fact that it doesn't simulate in transient is a cause for concern. As Shawn says, if it doesn't simulate in transient it won't in shooting PSS either. The convergence issue you showed appears during the initial transient (probably - because you sent screen shots of parts of the log - it would have been better if you'd attached the spectre.out file and also the input.scs to be sure what you're actually simulating). Maybe there's a problem with the circuit itself which is causing convergence failure - or maybe the inductors are making it unstable and it is taking short timesteps. Hard to know without seeing what you're actually simulating.
In general I would suggest using harmonic balance for this type of two-tone simulation. An LNA should be reasonably linear, and so a two-tone harmonic balance simulation (rather than trying to simulate the common frequency in shooting PSS) would be more efficient and more accurate (assuming you ask for a sufficient number of harmonics of each tone). That said, I'd be worried if I couldn't simulate it in transient...