SVR4.0.4 added the `Advanced Compatibility Package', which provides iBCS-2 compliance for running SVR3 binaries. These facilities are also present in SVR4.2. XFree86 makes use of this to accept local connections from SVR3 clients. The XFree86 binary distribution is built to use these capabilities. You need to install the `Advanced Compatibility Package', if you have not done so already.
We have found that SVR4.0.4 is not able to run all SCO, and perhaps not many ISC SVR3 binaries. This is not a failing of XFree86, but of SVR4 itself. One particular example is that many SVR3 programs are ignorant of the UFS filesystem, and attempt to read directories as files, rather than using the system call that is defined for the purpose. This will fail for obvious reasons. The SVR4.0.4 release notes from USL (which you should have gotten from your vendor) have lots of suggestions for how to improve compatibility.
That said, we have had luck with several SCO binaries right out of the box. No changes are needed - just go to an xterm window and run the program.
ISC users will need a binary editor before they can attempt to run their
binaries. ISC, for whatever reason, put the pipe for local connections in
/tmp/.X11-unix/Xn. This unfortunately is the same place as the X
Consortium X server puts the Unix-domain socket normally used for local
connections. The XFree86 server was modified to use
/dev/X/ISCCONN/Xn for
local connections to ISC clients. So what you must do is use a binary
editor to edit your client program. Search for /tmp/.X11-unix,
and change
it to /dev/X/ISCCONN. Now you just have to worry about base-OS
compatibility.