You cannot permanently remove mode groups from the Shortcut bar since these are special groups which are automatically selected when you change mode, but you can hide mode groups by clicking the right mouse button in a blank part of the Shortcut bar and selecting Mode Groups from the context menu.
Any groups that you have added may be deleted from the Shortcut bar by clicking on the relevant group tab using the right mouse button and selecting Remove Group from the context menu.