deRegApp doesn't work

I'm not sure what you expect from anyone here - you've given so little information. deRegApp does register in general - so please provide the details of what you've actually done - otherwise you're pretty unlikely to get an answer.

Andrew 

in open file form,add a custom application

deRegApp(?appName "testApp"

?enableTrigger 'etrig

?dataTrigger 'dtrig

?viewType "testLayout"

?parentType "maskLayout")

in open file form,no new app,in new file,new app exist

Support for applications having a more flexible "Open With" was enhanced in IC23.1 - before that, the apps had to end in L, XL, etc - see https://support.cadence.com/apex/ArticleAttachmentPortal?id=a1Od0000000nZI0EAM&pageName=ArticleContent

Now you can do:

deAddAppCapabilities("testLayout" ?openFile t)

Not clear what you're actually trying to do here, but maybe that's the missing piece. You're presumably not using IC23.1 though since "Layout EAD" is shown and it would be "Layout EXL" in IC23.1

Andrew

Thank you.