The gpgsmsk parameter specifies the mask used to determine if a given page may be swapped. Whenever the pager (vhand) is run, it decrements software reference bits for every active page. When a process subsequently references a page, the counter is reset to the limit (NDREF, as defined in /usr/include/sys/immu.h). When the pager is looking for pages to steal back (if memory is in short supply), it takes only pages whose reference counter has fallen to gpgsmsk or below.
This parameter is specified in /var/sysgen/mtune.
Also see /usr/include/sys/immu.h and /usr/include/sys/tuneable.h and the gpgshi and gpgslo kernel parameters for more information.