How can I assign ground shape 20 mil away from the outline using PCB editor 17.4?

you will need to make sure the DXF import created a closed shape.

Select the Design Outline shape, use the Zcopy command to paste it to a copper layer, shrink or expand by 20 mils in the options tab.

An alternate method you see in Altium translations is adding an etch keepout to that subclass.  But, the first method is usually easier.

How can I make sure it is a closed shape?, This is a useful link I have found: Unfortunately I can not find the compose shape option under the shape 

Hi Frank.

So there are a couple of things. First you want to have a "Board Outline" In allegro/orcad this is a "Shape" and it is on the board geometry Class > Outline or board outline sub class. Basically this is completely enclosed.

You also want a Route Keepin all shape inside that board outline. The route keepin all is on the Route Keepin class. The shape type will be unfilled.

Give this a try.

1
Import your DXF Outline and put it on the "Board Geometry" Class > "Assembly_Detail Class" Next go to the board geometry options and verify that your board geometry layer for the board geometry outline is turned on. In the find filter turn on everything so you can select objects.

2
The next step will create a shape out of that imported DXF. Go to edit Z-Copy. In the options pane choose "Board Geometry Outline or "Board Outline" depending on your Allegro Ver. Window select the DXF outline and right click done
to complete it.

3
To verify you now have a viable board outline turn off the Assembly Detail Layer/Class. You should see your Board outline now. If you hover the mouse over it's edge you should see it is a complete shape.

The operation to create your route keepin is the same except after Z-Copy your options will be "RouteKeepin" All class. So Z-COPY your DXF outline to the routekeepin Class. When you do that you will find a route keepin shape has
been created from your DXF as in step 2 from that outline that resides on the Assembly_Detail Layer-Class.

The last thing to do is Reduce the size of the newly created route keepin shape. To do this window select the shape. Right Click and choose Expand/Contract. In the options pane use the + or - buttons to reduce it by the size you want.

If you do everything correct you should see the routekeepin inside that board outline. If you decide to pour some copper later on inside the board outline it will not go beyond the route keepin. To get a hang for the mechanics of this it may be easier to turn off layers so you can easily see the newly created objects after using the z-copy command. Having different layer colours too can help.

Here is a pic to illustrate what it can look like. Notice the board outline in pink with the keepin inside of that and then a copper pour on the top layer that does not extend beyond the board outline.

All the best.

A problem with a lot of the tips out there refer to the "Legacy Menu" option where "Shape->Compose Shape" is.  You can set your menu system to legacy or you can use the operation "Shape->Create Shape from Lines" 

I have no idea why Cadence changed this but it is the same feature.  The legacy menu option is under the "Display" menu.  

Thank you for the detailed explanation. Fixed the issue.