I have a master layout and with various levels of heirarchy. My aim is to find and draw boxes around shapes belonging to a particular layer.
The problem comes in when I come across an instance which is a pcell while I am traversing through the hierarchy.
If I try to openthe instance using dbOpenCellViewByType(), it obviously opens with the default parameters and the purpose is lost.
I intend to do some processing on pcell layout with the instantiated parameters "without flatenning" the layout.
Do you intend that the new shape be put at the top level of the hierarchy?
Pcell instances create a virtual memory master that reflects the results of the execution of the pcell code based on the parameter values. For each set of parameter values, there is one instance master (instHeader). An instHeader may point to multiple masters.
You should not modify the VM instance master of the pcell. It can corrupt the VM master as well as the cached master.
Like Derek said, you would have to find the shape in the master and then draw a shape at the top level for each instance using that instance's transform. (I would use dbTransformBBox but I'm lazy)
For shapes nested in deeper levels of hierarchy, you would need to find the instance path and concat the transforms. See dbGetTrueOverlaps and dbConcatTransform for more information on this.