Ok I tried (see below), but resizing the screen after creating it is too late for -nomax (it determines window size before the first possible place the screen can be resized dynamically).
If you want to force a smaller window:
mame sdiamond -resolution 640x480
However, the user won't be able to make the window larger.
Code:
//(make sure to #include screen.h)
screen_device_enumerator screen_iter(root_device());
for (screen_device &screen : screen_iter)
{
if (screen.screen_type() == SCREEN_TYPE_SVG)
{
int new_width = screen.width() / 4;
int new_height = screen.height() / 4;
screen.configure(new_width, new_height, rectangle(0 , new_width - 1 , 0 , new_height - 1), screen.frame_period().as_attoseconds());
}
}