The size of the area used by the UAP trace facility is specified using the uap_trace_max operand in the user service definition.
See the manual OpenTP1 System Definition for details on the user service definition.