Flexible scaling
Flexible scaling gives you more choices for the apparent size of the desktop interface. BetterDisplay generates a fine-grained range of scaled resolutions that follow the display's native aspect ratio, giving the resolution slider more steps between its larger and smaller settings.
For the underlying rendering method, see what flexible scaling and custom resolutions change.
For a physical display, this belongs to System Configuration and Flexible Scaling. It uses system overrides, so editing a value is not the same as activating it: the Apply and activation/reboot requirements still matter. A virtual screen can instead generate its own resolution list.
These are different paths with different prerequisites. Neither removes macOS or hardware limits, even when the app lets you describe the resolution you want.
See Resolution, Aspect ratio and the system configuration notices. The System Configuration and Flexible Scaling page explains the settings and activation steps.