There is little point in running the X Window System on a server with no screen. Edit /etc/inittab finding the line containing initdefault, such as
id:5:initdefault:
Alter the default from run level 5 (multiuser with X Window System) to run level 3 (multiuser).
id:3:initdefault:
The startx command can be used if an occassional X Windows session is required upon an attached keyboard and monitor.