I noticed that the last part of my previous posting got lost.
Here is another attempt.
In my previous message I mentioned DECdtm under VAX/VMS. To answer your last question :
|> If so, is there any experimentation going into building a database
|> system that makes use of these transactions in such a way that both
|> files and database tables could be updated in a single transaction.
DECdtm can coordinate updates from different resource managers.
So yes, you can start a transaction involving both RMS (VAX/VMS flat file system) and Rdb/VMS (DEC's relational database), and all transaction semantics will apply.