Persisting and storing data in a DataStore

The DataStore component provides the main access point to high performance data caching and compact persistence. It uses a simple hierarchical file system which allows it to contain DataExpress DataSets, arbitrary files, and Java Objects. The DataStore component uses a single file to store one or more data streams. A DataStore file has a hierarchical directory structure that associates a name and various directory status with a particular data stream. Classes in the DataStore package will remember the rows fetched in a table, even after the application has been terminated and restarted. For an example of using a DataStore, see the tutorial.

The public data stream categories in a DataStore are:

  1. DataSet.

    This stream provides data caching and persistence for any DataExpress DataSet that extends from StorageDataSet (i.e. QueryDataSet, ProcedureDataSet, and TableDataSet). All semantics of StorageDataSet are supported including master-detail, picklists, constraints, calculated fields, sorting, filtering, and aggregation.

  2. File.

    There are createFile and openFile DataStore methods for creating and opening arbitrary file streams. These methods return a FileStream that extends the Java InputStream class with additional seek() and write() methods.

    Once data of type Object is in the DataStore, it is treated as a file stream. There are readObject and writeObject methods for reading and writing Java objects to a DataStore. These Objects are serialized into the DataStore using Java serialization. Java Objects may also be persisted in a DataStore by using a DataSet with a Column that has an Object data type.

The DataStore Directory can be accessed by calling the DataStore.getDirectoryDataSet method. This returns a DataSet with several columns for the name, data stream category, modification time, size, delete status, etc. The directory is maintained as a sorted list of file names.

Using a DataStore with StorageDataSets

Caching and persisting StorageDataSets in a DataStore is accomplished through two required property settings on a StorageDataSet called StorageDataSet.store and StorageDataSet.storeName. By default, all StorageDataSets use a MemoryStore if the StorageDataSet.store property is not set. Currently MemoryStore and DataStore are the only implementations for the store property. The StorageDataSet.storeName property is the unique name associated with this StorageDataSet in the DataStore directory.

To store and persist a DataSet in a DataStore,

  1. Select the application's Frame file in the Navigation pane. Select the Design tab of the Content pane to activate the UI Designer.

  2. Add a DataStore component from the Data Express tab of the Component palette to the Component tree.

  3. In the Inspector, select the fileName property of the DataStore. Use the Browse button to bring up the Open dialog and enter the DataStore file name. Click Open.

  4. Select the DataSet component in the Component tree.

  5. In the Inspector, set the store property of the DataSet to the DataStore component.

Tutorial: Off-line editing with DataStore

This tutorial provides the steps for creating an application that uses a DataStore component to enable off-line editing of data. A discussion of how changes are saved to the DataStore component and how changes are saved back to the SQL server database are discussed in Saving changes to the SQL database on the server. An expanded version of the application includes a "Connect" button and disables the navigator buttons that aren't available until you're connected to a SQL server. To create this application,
  1. Select File|Close All.

  2. Create a new application by selecting File|New from the menu and double-clicking the Application icon. Accept all defaults, except on Page 2 of 2 of the Application Wizard, check the Add Status Bar box.

  3. Select the Frame file in the Navigation pane. Select the Design tab of the Content pane. Click a Database component on the Data Express tab of the Component palette. Click in the Component tree to add the Database component to the application.

  4. Open the connection property editor for the Database component by double-clicking the connection property in the Inspector. Set the connection properties to the Local InterBase samples:
    Property name Value
    Connection URL jdbc:odbc:DataSet Tutorial
    Username SYSDBA
    Password masterkey

    Click the Test Connection button to check that the connection properties have been correctly set. Results of the connection attempt are displayed in the gray area below the button. When the connection is successful, click OK.

  5. Add a DataStore component from the Data Express tab to the Component tree.

    Adding a DataStore component writes an import statement for it to your code, and adds the "Datastore 2.0" library to your project properties.

    If you write code that instantiates a DataStore, instead of letting the Designer do it, you must add the import statement and the library. To add the library, select File|Project Properties, click the Add button, and choose the "Datastore 2.0" library.

  6. In the Inspector, select the fileName property of the DataStore. Use the Browse button to bring up the Open dialog and enter the DataStore file name. You do not have to specify a file extension; a DataStore always has the extension .jds. Click Open.

  7. Add a QueryDataSet component to the Designer by clicking on the QueryDataSet component on the Data Express tab and then clicking in the Component tree.

    Double-click the query property of the QueryDataSet component in the Inspector and set the following properties:
    Property name Value
    Database database1
    SQL Statement select * from employee
    Place SQL text in resource bundle unchecked

    Click Test query to ensure that the query is runnable. When the gray area beneath the button indicates Success, click OK to close the dialog.

  8. Select the store property for the QueryDataSet. Select dataStore1 from the list.

  9. Select the storeName property in the Inspector. Enter the name employeeData.

  10. Select the this component in the UI section of the Component tree. Click the Events tab in the Inspector. Select the windowClosing event, and double-click to create an event listener and the skeleton of the method to be called when the event occurs. Add this code to the method:
        try {
          if (dataStore1.isOpen()) {
            dataStore1.close();}
        }
        catch (Exception ex) {
          ex.printStackTrace();
        }
    

  11. Click the Design tab to return to design mode. Select the bevelPanel1 component in the UI section of the Component tree. Click the NavigatorControl on the JBCL tab, then click in the BevelPanel in the Designer. Set its dataSet property to queryDataSet1.

  12. Select the BevelPanel in the Component tree. In the Inspector, set its layout property to FlowLayout, and set its constraints property to North.

  13. Add a GridControl by clicking on the component in the Component palette, and drawing the grid to the proper size in the Design window. In the Inspector, set its constaints property to Center, and set its dataSet property to queryDataSet1.

  14. In the Component tree, select the StatusBar (this component was added in the Application wizard). In the Inspector, set its constraints property to South, and set its dataSet property to queryDataSet1.

    Note: A DataStore's create() method is different from other components. The create() method must be called after its properties are set and before it is opened. Setting properties after a DataStore file is created has no effect. The create() method writes an empty DataStore to the location and name given by the fileName property. When you define and use a DataStore in the UI Designer, the create operation will be done for you at the point where the DataStore is opened. A line of code that calls create() is not written to your code, because you wouldn't want it to be executed each time the application is run. If you write code to use a DataStore manually, you should include code such as:

        if ( ! (new File("C:\\mypath\\myDataStore.jds").exists()))
          dataStore1.create();
    
  15. Select Run|Run to compile and run the application.

In the running application, make some changes to the data and click the Post button on the navigator to save the changes to the DataStore file. Changes are also saved to the file when you move off of a row, just as they are with an in-memory data set (MemoryStore). To save changes back to the data source, see Saving changes to the SQL database on the server.

Note: A data set in a DataStore can have tens or hundreds of thousands of rows. Handling that much data using an in-memory data set would probably result in some unsatisfactory performance issues.

Close the application and run it again. You see the data as you edited it in the previous run. This, of course, is very different from the behavior of an in-memory data set. If you want, you can exit the application, shut down Local InterBase Server (by right-clicking on its icon in the task bar and choosing Shutdown), and run the application again. Without any connection to the SQL database, you can continue to view and edit data in the DataStore. This would be especially useful if you want to work with data off-line, such as at home or on an airplane.

Saving changes to the SQL database on the server

So far in the tutorial, nothing has been saved back to the SQL database on the server. On the NavigatorControl,

Options set in the queryDescriptor also have an effect on how data is stored, saved, and refreshed. In the queryDescriptor in this example, the Execute Query Immediately When Opened option is selected. This option indicates how data is loaded into the DataStore file when the application was first run. On subsequent runs, the execution of the query is suppressed, because the data set is found in the DataStore file instead of on the server. As a result,

The Execute Query Immediately When Opened option is not allowed to overwrite existing data. This means that you can safely close and reopen a data set to change property settings in either a MemoryStore or in a DataStore without losing editing changes.

Some other information related to working with the DataStore:

Expanding the tutorial to include a connection option

Once you've got data in the DataStore file, you can run this application and edit data whether Local InterBase Server is available or not. When you are working off-line, you have to remember not to click the navigator's Save or Refresh button. This topic expands the application to include a "Connect" button and disable the navigator buttons that are only available when a connection is established. To add this capability,

  1. If not already selected, select the Frame file in the Navigation pane, then select the Design tab.

  2. Click a ButtonControl on the JBCL tab, and click at the top of the Design window to place the component. In the Inspector, set its label property to Connect.

  3. In the Inspector, click the Events tab, select the actionPerformed event for the button, and double-click to locate the position to enter this code into the skeleton of the method:
        try {
          if (!connected) {
            // Connect to SQL database
            database1.openConnection();
            navigatorControl1.setButtonEnabled(NavigatorControl.SAVE, true);
            navigatorControl1.setButtonEnabled(NavigatorControl.REFRESH, true);
            buttonControl1.setLabel("Disconnect");
            connected = true;
          }
          else {
            // Disconnect from SQL database
            database1.closeConnection();
            navigatorControl1.setButtonEnabled(NavigatorControl.SAVE, false);
            navigatorControl1.setButtonEnabled(NavigatorControl.REFRESH, false);
            buttonControl1.setLabel("  Connect  ");
            connected = false;
          }
        }
        catch (Exception ex) {
          ex.printStackTrace();
        }
    

  4. In the same Source window, define this variable near the top of the class definition, before the constructor,
      boolean connected;        // connection made to SQL database?
    

  5. Click the Design tab, select the this component, select the Events tab, select the windowOpened event, and double-click to add this initialization to the generated event handler skeleton:
        // Disable Navigator buttons that require SQL database until connection is made
        navigatorControl1.setButtonEnabled(NavigatorControl.SAVE, false);
        navigatorControl1.setButtonEnabled(NavigatorControl.REFRESH, false);
    

  6. Run the application.

Now, the navigator's Save and Refresh buttons are dimmed, clicking them has no effect. When a connection is successfully established by clicking the Connect button, the button's label will change to "Disconnect" and the navigator's Save and Refresh buttons will be enabled.

Usage recommendations

It is strongly recommended that DataStore components be placed inside data modules and that all StorageDataSets that use a particular DataStore be placed in the same DataModule that the DataStore is placed in. Any references to these StorageDataSets should be made through DataModule accesser methods such as businessModule.getCustomer(). The reason for this is that much of the functionality surfaced through StorageDataSets is driven by property and event settings. Although most of the important "structural" StorageDataSet properties are persisted in the DataStore directory, the classes that implement the event listener interfaces are not. By instantiating the StorageDataSet with all event listener settings, constraints, calc fields, and filters implemented with events, they will be properly maintained at run time and design time.

Be sure to call DataStore.close when the DataStore is no longer needed. Closing a DataStore ensures that all modifications are saved to disk. There is a daemon thread for all open DataStore instances that is constantly saving modified cache data (by default modified data is saved every 100 milliseconds). Although the DataStore daemon is very good at saving off DataStore modifications, there are some precautions that should be taken:

  1. Do not call System.exit() until you have called close on all open DataStore components.

  2. Use copies of your DataStore files when debugging. The debugger stops all threads when single-stepping or when breakpoints are hit. This keeps the DataStore daemon thread from saving modifed cache data.

If you suspect that cache contents were not properly saved on a DataStore, there is a borland.datastore.StreamVerifier class with public static verify() methods that can verify a single stream or all streams in a DataStore.

Another benefit to closing DataStore components is that when all DataStore components are closed, the memory allocated to the DataStore cache is released.

Close all DataSets that have their store property set to a DataStore when you are done with them. This frees up DataStore resources associated with the DataSet and allows the DataSet to be garbage collected.

Restructuring DataStore StorageDataSets

The persistent column editor in the JBuilder UI Designer provides support for moving, deleting, and inserting columns. Column data types can also be changed. To activate the persistent column editor,

  1. Select the Data Module file in the Navigation pane.
  2. Press the Design tab for your DataModule.
  3. Right-click on a StorageDataSet in the JBuilder Structure pane.
  4. Select Activate Designer.

This designer works for StorageDataSets that are using a MemoryStore or a DataStore. MemoryStore performs all operations instantly. When a Column data type is changed, MemoryStore does not convert data values to the new data type. The old values are lost.

DataStore does not perform the move/insert/delete/change type operations on StorageDataSets immediately. The structural change is noted inside the DataStore directory as a "pending" operation. The StorageDataSet.getNeedsRestructure method returns true when there is a "pending" restructure operation. A StorageDataSet with pending structural changes can still be used.

To make a pending restructure operation take effect, press the restructure speed bar button in the persistent column editor You can also cause a restructure operation to happen programmatically by calling the StorageDataSet.restructure() method.

The restructure method can be used even when there are no pending structural changes to repair or compact a StorageDataSet and its associated indexes.

Persistent column editing

Many DataSets, like QueryDataSet and ProcedureDataSet, obtain their structure and row data from a SQL server. But there is also the need to add columns to these DataSets or to create stand-alone tables. This can be accomplished with the JBuilder persistent column editor. DataSet columns can be changed, deleted, moved, and added instantly at design time. The DataStore uses a mapping table to provide an illusion that each structural operation has occurred. When all structural changes are complete, the Restructure button can be pressed

When the UI Designer is activated on while a StorageDataSet node is selected, the columns for that node are displayed in a grid on the design surface. A toolbar for adding, deleting, moving up, moving down, and restructuring the data set is displayed above the grid.

When to use a DataStore

Undeleting DataStore streams

Streams are deleted in a DataStore by marking their directory entry as deleted. The contents of these streams will be reallocated as new storage is needed for existing streams. Although streams can be undeleted, the entire contents may not be present if space was needed for new allocations.

Capacity

The maximum DataStore file size: Blocksize * 2 gigabytes. The default block size is 2k, so that is about 4,400 gigabytes. The maximum rows per DataSet is 4 gig. The maximum row length is 1/3 block size, but long values are stored as blobs (blobs have a 2 gigabyte limit).