It is not possible to have privacy for some users.
You could use panel contexts (tenants) to have two separate button config files, but you will have to use custom configuration for that and not use the fop2 manager. One tenant can have privacy enabled, and a different tenant privacy not enabled.
Quite a bit of work, but it would be doable that way.