With the exception of the pseudo terminal (pty) device, which only works on Linux AFAIK, all the RS-232 devices in MAME are internal.

How to set it up on the (emulated) chess computer: differs per chess computer, I'd suggest to refer to the manual of the one you're playing with: http://www.zanchetta.net/default.asp...documentations
Protocol: differs per company (Novag Super System, Saitek OSA, Fidelity).

I've only tested it with the terminal device myself. For example:
mame.exe diablo68 -numscreens 2 -rs232 terminal
Go to video options, select chesscomputer view on one window, and terminal view on the other. Go to input settings, keyboard selection, and disable the root keyboard so you can input commands on the terminal. Then for example, press P, Enter in the terminal window to see the current chess position.