<< Previous | Index | Next >>

10. The Flash File System

Dynamic C 7.0 introduced a simple file system that can be used with a second flash memory or in SRAM. Dynamic C 7.05 introduced an improved file system with more features:

This file system, known as the filesystem mk II or simply as FS2, uses the same API as the first file system, with some additional functions. Initialization is performed slightly differently, and the data format is not compatible. Z-World recommends that FS2 be used for all new applications. The first file system, which we will refer to as FS1, will be maintained but enhancements will only be implemented for FS2.

The Dynamic C file system supports a total of 255 files. Unlike FS1, it is not possible to reserve a range of file numbers for system use with FS2. Equivalent functionality is available via partitioning of devices.

The low-level flash memory access functions should not be used in the same area of the flash where the flash file system exists.

10.1 General Usage

The recommended use of a flash file system is for infrequently changing data or data rates that have writes on the order of tens of minutes instead of seconds. Rapidly writing data to the flash could result in using up its write cycles too quickly. For example, consider a 256K flash with 64 blocks of 4K each. Using a flash with a maximum recommendation of 10,000 write cycles means a limit of 640,000 writes to the file system. If you are performing one write to the flash per second, in a little over a week you will use up its recommended lifetime.

Increase the useful lifetime and performance of the flash by buffering data before writing it to the flash. Accumulating 1000 single byte writes into one can extend the life of the flash by an average of 750 times. FS2 does not currently perform any in-memory buffering. If you write a single byte to a file, that byte will cause write activity on the device. This ensures that data is written to non-volatile storage as soon as possible. Buffering may be implemented within the application if possible loss of data is tolerable.

NOTE The use of USE_2NDFLASH_CODE is not compatible with the flash file system.

10.1.1 Maximum File Size

The maximum file size for an individual file depends on the total file system size and the number of files present. Each file requires at least two sectors: at least one for data and always one for metadata (for information used internally). There also needs to be two free sectors to allow for moving data around. It is not recommended to use the flash file system to store a large number of small files. It is much more efficient to have a few large ones.

10.1.2 Using SRAM

The flash file system can be used with battery-backed SRAM. Internally, RAM is treated like a flash device, except that there is no write-cycle limitation, and access is much faster. The file system will work without the battery backup, but would, of course, lose all data when the power went off.

Currently, the maximum size file system supported in RAM is about 200k. This limitation holds true even on boards with a 512k RAM chip. The limitation involves the placement of BIOS control blocks in the upper part of the lower 256k portion of RAM.

To obtain more RAM memory, xalloc() may be used. If xalloc() is called first thing in the program, the same memory addresses will always be returned. This can be used to store non-volatile data is so desired (if the RAM is battery-backed), however, it is not possible to manage this area using the file system.

When using FS1, since only one device type is allowed at a time, the entire file system would have to be in SRAM. This is recommended for debugging purposes only. Using FS2 increases flexibility, with its capacity to use multiple device types simultaneously. Since RAM is usually a scarce resource, it can be used together with flash memory devices to obtain the best balance of speed, performance and capacity.

10.1.3 Wear Leveling

The current code has a rudimentary form of wear leveling. When you write into an existing block it selects a free block with the least number of writes. The file system routines copy the old block into the new block adding in the users new data. This has the effect of evening the wear if there is a reasonable turnover in the flash files.

10.1.4 Low-Level Implementation

For information on the low-level implementation of the flash file system, refer to the beginning of the library files FS2.LIB and FS_DEV.LIB if using FS2, or library file FILESYSTEM.LIB, if using FS1.

10.1.5 Multitasking and the File System

Neither FS1 nor FS2 are re-entrant. If using preemptive multitasking, ensure that only one thread performs calls to the file system, or implement locking around each call.

10.2 Application Requirements

The application requirements for FS1 and FS2 are slightly different. This section covers both sets of requirements, including:

10.2.1 FS1 Requirements

To use the file system, a macro that determines which low-level driver is loaded must be defined in the application program. 


#define FS_FLASH   // use 2nd flash for file system
#define FS_RAM     // use SRAM (supported for debug purposes)

The file system library must be compiled with the application. 


#use "FILESYSTEM.LIB"

10.2.2 FS1 and Use of the First Flash

To use FS1 in the first flash, a low-level driver must be used: 


#define FS_FLASH_SINGLE

Because this particular low-level driver must share the first flash with the program code, the file system must be carefully placed such that the two do not collide. Also, it should be noted that any time the first flash is written to during runtime, interrupts will be shut off for the duration of the write. This could have serious implications for real-time systems.

To reserve space in the first flash, such that Dynamic C will not clobber the file system, a minor BIOS modification is necessary. The macro XMEM_RESERVE_SIZE in the BIOS is currently set to 0x0000. Increasing this value will reserve that much space between the end of xmem code that Dynamic C is building, and the System ID block at the end of memory. Unfortunately, the file system needs to start on a FS_BLOCK_SIZE boundary, which is normally 4096 bytes. Therefore, slightly more space than is needed should be allocated, to allow for the System ID block and that the end of xmem space might not lie on a 4096 byte boundary.

After this space has been allocated, the beginning of the file system can be found. The end of where Dynamic C will touch the flash is stored in the macro END_OF_XMEMORY, and the file system may start at the next 4096 byte boundary after that point. The following code computes what to pass to fs_format(). 


// where to start the file system
long fs_start;

// start at the end of xmem 
fs_start = END_OF_XMEMORY;

// divide out the blocksize, to meet requirements for fs_format
fs_start = fs_start / FS_BLOCK_SIZE;

if((fs_start * FS_BLOCK_SIZE) != END_OF_XMEMORY)
{
   // rounding error: move up 1 block so end of xmem is not clobbered
   fs_start++;

}
fs_format(fs_start, NUM_BLOCKS, 0);

After this point, the file system should act normally.

If the 4096 byte block size is too large, given the limited room in the first flash, that can be overwritten with the macro: 


#define FS_BLOCK_SIZE 512

See the sample program, 1stflash.c, for an example of using the first flash with FS1.

10.2.3 FS2 Requirements

The file system library must be compiled with the application: 


#use "FS2.LIB"

For the simplest applications, this is all that is necessary for configuration. For more complex applications, there are several other macro definitions that may be used before the inclusion of FS2.LIB. These are: 


#define FS_MAX_DEVICES   3
#define FS_MAX_LX        4
#define FS_MAX_FILES    10

These specify certain static array sizes that allow control over the amount of root data space taken by FS2. If you are using only one flash device (and possibly battery-backed RAM), and are not using partitions, then there is no need to set FS_MAX_DEVICES or FS_MAX_LX.

For more information on partitioning, please see section 10.4, "Setting up and Partitioning the File System," on page 107.

10.2.4 FS2 Configuration Macros

FS_MAX_DEVICES

This macro defines the maximum physical media. If it is not defined in the program code, FS_MAX_DEVICES will default to 1, 2, or 3, depending on the values of FS2_USE_PROGRAM_FLASH, XMEM_RESERVE_SIZE and FS2_RAM_RESERVE.

FS_MAX_LX

This macro defines the maximum logical extents. You must increase this value by 1 for each new partition your application creates. It this is not defined in the program code it will default to FS_MAX_DEVICES.

For a description of logical extents please see section 10.4.2, "Logical Extents (LX)," on page 108.

FS_MAX_FILES

This macro is used to specify the maximum number of files that are allowed to coexist in the entire file system. Most applications will have a fixed number of files defined, so this parameter can be set to that number to avoid wasting root data memory. The default is 6 files. The maximum value for this parameter is 255.

FS2_RAM_RESERVE

This BIOS-defined macro determines the amount of space used for FS2 in RAM. If some battery-backed RAM is to be used by FS2, then this macro must be modified to specify the amount of RAM to reserve. The memory is reserved near the top of RAM. Note that this RAM will be reserved whether or not the application actually uses FS2.

Prior to Dynamic C 7.06 this macro was defined as the number of bytes to reserve and had to be a multiple of 4096. It is now defined as the number of blocks to reserve, with each block being 4096 bytes.

FS2_USE_PROGRAM_FLASH

The number of kilobytes reserved in the first flash for use by FS2. The default is zero. The actual amount of flash used by FS2 is determined by the minimum of this macro and XMEM_RESERVE_SIZE.

The first flash may be used in FS1as well. See section 10.2.2 for details.

XMEM_RESERVE_SIZE

This BIOS-defined macro is the number of bytes (which must be a multiple of 4096) reserved in the first flash for use by FS2 and possibly other customer-defined purposes. This is defined in the BIOS as 0x0000. Memory set aside with XMEM_RESERVE_SIZE will NOT be available for xmem code.

10.2.5 FS2 and Use of the First Flash

To use the first flash in FS2, follow these steps:

  1. Define XMEM_RESERVE_SIZE (currently set to 0x0000 in the BIOS) to the number of bytes to allocate in the first flash for the file system.

  2. Define FS2_USE_PROGRAM_FLASH to the number of KB (1024 bytes) to allocate in the first flash for the file system. Do this in the application code before #use "fs2.lib".

  3. Obtain the LX number of the first flash: Call fs_get_other_lx()when there are two flash memories; call fs_get_flash_lx() when there is only one.

  4. If desired, create additional logical extents by calling the FS2 function fs_setup() to further partition the device. This function can also change the logical sector sizes of an extent. Please see the function description for fs_setup() in the Dynamic C Function Reference Manual for more information.

10.2.5.1 Example Code Using First Flash in FS2

If the target board has two flash memories, the following code will cause the file system to use the first flash: 


FSLXnum flash1;                   // logical extent number
File f;                          // struct for file information

flash1 = fs_get_other_lx();
if (flash1) {
   fs_set_lx(flash1, flash1);
   fcreate(&f, 10);
   . . .
}

To obtain the logical extent number for a one flash board, fs_get_flash_lx() must be called instead of fs_get_other_lx().

10.3 Functions

For backwards compatibility FS2 uses the same function names as FS1. Some functions have enhanced semantics when using FS2. For example fwrite() will allow writing over existing parts of the file rather than just appending.

10.3.1 FS1 API

These functions are the file system API for FS1. They are defined in FILESYSTEM.LIB. For a complete description of these functions please see the Dynamic C Function Reference Manual.

Table 10-15. FS1 API
Command
Description
fs_init (FS1) Initialize the internal data structures for the file system.
fs_format (FS1) Initialize the flash memory and the internal data structures.
fs_reserve_blocks (FS1) Reserves blocks for privileged files.
fsck (FS1) Verifies data integrity of files.
fcreate (FS1) Creates a file and open it for writing.
fcreate_unused (FS1) Creates a file with an unused file number.
fopen_rd (FS1) Opens a file for reading.
fopen_wr (FS1) Opens a file for writing (also opens it for reading.)
fshift Removes specified number of bytes from file.
fwrite (FS1) Writes to the end of a file.
fread (FS1) Reads from the current file pointer.
fseek (FS1) Moves the read pointer.
ftell (FS1) Returns the current offset of the file pointer.
fclose Closes a file.
fdelete (FS1) Deletes a file.

10.3.1.1 FS1 API Details

The functions fs_init and fs_format are similar, in that they both start the file system. Use fs_format() to erase all blocks in the file system. This function's third parameter, wearlevel, should be 1 for a new flash memory; otherwise it should be 0 to use the current wear leveling.

Use fs_init() to preserve blocks that are in use and to do an integrity check of them. In case of loss of power, fs_init() will delete any blocks that may be partially written and will substitute the last known good block for that file. This means that any changes to the file that occurred between the last write and the power outage would be lost.

10.3.2 FS2 API

The API for FS2 is defined in FS2.LIB. For more information please see the Dynamic C Function Reference Manual.

Table 10-16. FS2 API
Command
Description
fs_setup (FS2) Alters the initial default configuration.
fs_init (FS2) Initialize the internal data structures for the file system.
fs_format (FS2) Initialize flash and the internal data structures.
lx_format Formats a specified logical extent (LX).
fs_set_lx (FS2) Sets the default LX numbers for file creation.
fs_get_lx (FS2) Returns the current LX number for file creation.
fcreate (FS2) Creates a file and open it for writing.
fcreate_unused (FS2) Creates a file with an unused file number.
fopen_rd (FS2) Opens a file for reading.
fopen_wr (FS2) Opens a file for writing (and reading).
fshift Removes specified number of bytes from file.
fwrite (FS2) Writes to a file starting at "current position."
fread (FS2) Reads from the current file pointer.
fseek (FS2) Moves the read/write pointer.
ftell (FS2) Returns the current offset of the file pointer.
fs_sync (FS2) Flushes any buffers retained in RAM to the underlying hardware device.
fflush (FS2) Flushes buffers retained in RAM and associated with the specified file to the underlying hardware device.
fs_get_flash_lx (FS2) Returns the LX number of the preferred flash device (the 2nd flash if available).
fs_get_lx_size (FS2) Returns the number of bytes of the specified LX.
fs_get_other_lx (FS2) Returns LX # of the non-preferred flash (usually the first flash).
fs_get_ram_lx (FS2) Return the LX number of the RAM file system device.
fclose Closes a file.
fdelete (FS2) Deletes a file.

10.3.2.1 FS2 API Details

The functions fs_init and fs_format are used in a slightly different manner than in FS1. fs_init() does not use its two parameters (reserveblocks and numblocks) since it computes appropriate values internally. fs_format() should only be called after fs_init(), if necessary. This function's first parameter, reserveblocks, must be 0; anything else returns an error. This is one of the few cases of incompatibility between FS1 and FS2. The third parameter, wearlevel, should be 1 for a new flash memory; otherwise it should be 0 to use the current wear leveling.

The fsck() function is not available and is not needed in FS2; fs_init() always completely checks for internal consistency.

Refer to \Samples\FileSystem\FS2DEMO1.C for more details.

10.3.2.2 FS2 API Error Codes

When an API function returns an error, it may also return an error code in the global variable errno. The error codes are defined in the library file ERRNO.LIB.

10.4 Setting up and Partitioning the File System

FS2 can be more complex to initialize than FS1. This is because multiple device types can be used in the same application. For example, if the target board contains both battery-backed SRAM and a second flash chip, then both types of storage may be used for their respective advantages. The SRAM might be used for a small application configuration file that changes frequently, and the flash used for a large log file.

FS2 automatically detects the second flash device (if any) and will also use any SRAM set aside for the file system (if FS2_RAM_RESERVE is set).

10.4.1 Initial Formatting

The filesystem must be formatted when it is first used. The only exception is when a flash memory device is known to be completely erased, which is the normal condition on receipt from the factory. If the device contains random data, then formatting is required to avoid the possibility of some sectors being permanently locked out of use.

Formatting is also required if any of the logical extent parameters are changed, such as changing the logical sector size or re-partitioning. This would normally happen only during application development.

The question for application developers is how to code the application so that it formats the filesystem only the first time it is run. There are several approaches that may be taken:

10.4.2 Logical Extents (LX)

In FS2, the presence of both "devices" causes an initial default configuration of two logical extents to be set up. An LX is analogous to disk partitions used in other operating systems. It represents a contiguous area of the device set aside for file system operations. An LX contains sectors that are all the same size, and all contiguously addressable within the one device. Thus a flash device with three different sector sizes would necessitate at least three logical extents, and more if the same-sized sectors were not adjacent.

FS1 does not allow mixing of devices; it supports only one LX as defined in this document.

Files stored by the file system are comprised of two parts: one part contains the actual application data, and the other is a fixed size area used to contain data controlled by the file system in order to track the file status. This second area, called metadata, is analogous to a "directory entry" of other operating systems. The metadata consumes one sector per file.

The data and metadata for a file are usually stored in the same LX, however they may be separated for performance reasons. Since the metadata needs to be updated for each write operation, it is often advantageous to store the metadata in battery-backed SRAM with the bulk of the data on a flash device.

10.4.2.1 Specifying Logical Extents

When a file is created, the logical extent(s) to use for the file are defined. This association remains until the file is deleted. The default LX for both data and metadata is the flash device (LX #1) if it exists; otherwise the RAM LX. If both flash and RAM are available, LX #1 is the flash device and LX #2 is the RAM.

When creating a file, the associated logical extents for the data and the metadata can be changed from the default by calling fs_set_lx(). This functions takes two parameters, one to specify the LX for the metadata and the other to specify the LX for the data. Thereafter, all created files are associated with the specified LXs until a new call to fs_set_lx() is made. Typically, there will be a call to fs_set_lx() before each file is created, in order to ensure that the new file gets created with the desired associations. The file creation function, fcreate(), may be used to specify the LX for the metadata by providing a valid LX number in the high byte of the function's second parameter. This will override any LX number set for the metadata in fs_set_lx().

10.4.2.1.1 Further Partitioning

FS2 allows the initial default logical extents to be divided further. This must be done before calling fs_init(). The function to create sub-partitions is called fs_setup(). This function takes an existing LX number, divides that LX according to the given parameters, and returns a newly created LX number. The original partition still exists, but is smaller because of the division. For example, in a system with LX#1 as a flash device of 256K and LX#2 as 4K of RAM, an initial call to fs_setup() might be made to partition LX#1 into two equal sized extents of 128K each. LX#1 would then be 128K (the first half of the flash) and LX#3 would be 128K (the other half). LX#2 is untouched.

Having partitioned once, fs_setup() may be called again to perform further subdivision. This may be done on any of the original or new extents. Each call to fs_setup() in partitioning mode increases the total number of logical extents. You will need to make sure that FS_MAX_LX is defined to a high enough value that the LX array size is not exceeded.

While developing an application, you might need to adjust partitioning parameters. If any parameter is changed, FS2 will probably not recognize data written using the previous parameters. This problem is common to most operating systems. The "solution" is to save any desired files to outside the file system before changing its organization; then after the change, force a format of the file system.

Note that in particular, files written by FS1 are not readable by FS2 since the two file systems are incompatible at the device level.

10.4.3 Logical Sector Size

fs_setup() can also be used to specify non-default logical sector (LS) sizes and other parameters. FS1 uses fixed logical sectors (i.e. "blocks") of 4096 bytes. FS2 allows any LS size between 64 and 8192 bytes, providing the LS size is an exact power of 2. Each LX, including sub-partitions, can have a different LS size. This allows some performance optimization. Small LSs are better for a RAM LX, since it minimizes wasted space without incurring a performance penalty. Larger LSs are better for bulk data such as logs. If the flash physical sector size (i.e. the actual hardware sector size) is large, it is better to use a correspondingly large LS size. This is especially the case for byte-writable devices. Large LSs should also be used for large LXs. This minimizes the amount of time needed to initialize the file system and access large files. As a rule of thumb, there should be no more than 1024 LSs in any LX. The ideal LS size for RAM (which is the default) is 128 bytes. 256 or 512 can also be reasonable values for some applications that have a lot of spare RAM.

Sector-writable flash devices require: LS size ≥ PS size. Byte-writable devices, however, may use any allowable logical sector size, regardless of the physical sector size.

Sample program Samples\FileSystem\FS2DEMO2 illustrates use of fs_setup(). This sample also allows you to experiment with various file system settings to obtain the best performance.

FS2 has been designed to be extensible in order to work with future flash and other non-volatile storage devices. Writing and installing custom low-level device drivers is beyond the scope of this document, however see FS2.LIB and FS_DEV.LIB for hints.

10.5 File Identifiers

There are two ways to identify a particular file in the file system: file numbers and file names.

10.5.1 File Numbers

The file number uniquely identifies a file within a logical extent. File numbers must be unique within the entire file system. FS2 accepts file numbers in word format rather than the byte format of FS1: 


typedef word FileNumber

The low-order byte specifies the file number and the high-order byte specifies the LX number of the metadata (1 through number of LXs). If the high-order byte is zero, then a suitable "default" LX will be located by the file system. The default LX will default to 1, but will be settable via a #define, for file creation. For existing files, a high-order byte of zero will cause the file system to search for the LX that contains the file. This will require no or minimal changes to existing customer code.

Only the metadata LX may be specified in the file number. This is called a "fully-qualified" file number (FQFN). The LX number always applies to the file metadata. The data can reside on a different LX, however this is always determined by FS2 once the file has been created.

10.5.2 File Names

There are several functions in ZSERVER.LIB that can be used to associate a descriptive name with a file. The file must exist in the flash file system before using the auxiliary functions listed in the following table. These functions were originally intended for use with an HTTP or FTP server, so some of them take a parameter called servermask. To use these functions for file naming purposes only, this parameter should be SERVER_USER.

For a detailed description of these functions please refer to the Dynamic C's TCP/IP User's Manual, or use <Ctrl-H> in Dynamic C to use the Library Lookup feature.

Table 10-17. Flash File System Auxiliary Functions
Command
Description
sspec_addfsfile Associate a name with the flash file system file number. The return value is an index into an array of structures associated with the named files.
sspec_readfile Read a file represented by the return value of sspec_addfsfile into a buffer.
sspec_getlength Get the length (number of bytes) of the file.
sspec_getfileloc Get the file system file number (1- 255). Cast return value to FILENUMBER.
sspec_findname Find the index into the array of structures associated with named files of the file that has the specified name.
sspec_getfiletype Get file type. For flash file system files this value will be SSPEC_FSFILE.
sspec_findnextfile Find the next named file in the flash file system, at or following the specified index, and return the index of the file.
sspec_remove Remove the file name association.
sspec_save Saves to the flash file system the array of structures that reference the named files in the flash file system.
sspec_restore Restores the array of structures that reference the named files in the flash file system.

10.6 Skeleton Program Using FS1

The following program uses many of the file system commands. It writes several strings into a file, reads the file back and prints the contents to the STDIO window. The macro RESERVE should be 0 when the file system is in SRAM. When the file system is in flash memory you can adjust where it starts by defining RESERVE to be 0 or a multiple of the block size. 


#define FS_FLASH
#use "FILESYSTEM.LIB"

#define FORMAT
#define RESERVE  0L
#define BLOCKS  64
#define TESTFILE 1

main()
{
   File file;
   static char buffer[256];

#ifdef FORMAT
   fs_format(RESERVE,BLOCKS,1);
   if(fcreate(&file,TESTFILE)) {
      printf("error creating TESTFILE\n");
      return -1;
   }
#else
   fs_init(RESERVE,BLOCKS);
   if(fopen_wr(&file,TESTFILE) {
      printf("error opening TESTFILE\n");
      return -1;
   }
#endif
   fwrite(&file,"hello",6);
   fwrite(&file,"12345",6);
   fwrite(&file,"67890",6);

   while(fread(&file,buffer,6)>0) {
      printf("%s\n",buffer);
   }
   fclose(&file);
}

After running this program at least once, comment out "#define FORMAT." You will see that it runs in a similar fashion, but now the file is appended using fopen_wr() instead of being erased by fs_format() and then recreated with fcreate().

For a more robust program, more error checking should be included.

10.7 Skeleton Program Using FS2

The following program uses some of the FS2 API. It writes several strings into a file, reads the file back and prints the contents to the STDIO window. 


#use "FS2.LIB"
#define TESTFILE 1

main()
{
   File file;
   static char buffer[256];

   fs_init(0, 0);

   if (!fcreate(&file, TESTFILE) && fopen_wr(&file,TESTFILE))
   {
      printf("error opening TESTFILE %d\n", errno);
      return -1;
   }

   fseek(&file, 0, SEEK_END);
   fwrite(&file,"hello",6);
   fwrite(&file,"12345",6);
   fwrite(&file,"67890",6);
   fseek(&file, 0, SEEK_SET);

   while(fread(&file,buffer,6)>0) {
      printf("%s\n",buffer);
   }

   fclose(&file);

}

For a more robust program, more error checking should be included. See the sample programs in the \SAMPLES\FILESYSTEM folder for more complex examples which include error checking, formatting, partitioning and other new features.

FS2 returns more information in the case of errors than FS1. The library ERRNO.LIB contains a list of all possible error codes returnable by the FS2 API. These error codes mostly conform to POSIX standards. If the return value of an FS2 API indicates an error, then the errno variable may be examined to determine a more specific reason for the failure. The possible errno codes returned from each function are documented with the function.

<< Previous | Index | Next >>
Z-World, Inc.
www.zworld.com
Phone: 1.530.757.3737
Fax: 1.530.757.3792		 Rabbit Semiconductor
www.rabbitsemiconductor.com
Phone: 1.530.757.8400
Fax: 1.530.757.8402