Consider These Issues Before Building Your Own Filesystem
From personal computers to embedded devices, storing data for later is an essential part of most modern system. When storing that data, most often a filesystem (FS) is must-have for organizing that data to help find what you are looking for later.
Usually the filesystem is provided by the operating system, so applications don’t have to worry about it. Some examples of this are the Linux filesystems ext4, or the windows filesystems NTFS. But maybe you are building an operating system? Or running a device without one? Then you have a choice:
- Do you use a commercial off the shelf (COTS) FS?
- Do you develop your own?
- You may not need to. There are many robust already developed filesystems available for use, catering to many needs.
- Development time: it takes much less time to just use an existing FS than develop and whole new one.
- Support: COTS filesystems have creators and users who can answer your questions and help you with issues you may run into.
- Reliability: COTS filesystems have already been tested and have been running on many systems. If you build your own filesystem without expert guidance, bugs may show up in strange corner cases after weeks or months of use.
Despite these advantages of a COTS FS, there are other circumstances where building your own may make more sense:
- Requirements: You may have requirements for your filesystem that are not met by COTS filesystem.
- Performance: Writing your own filesystem may allow you to optimize it for your situation. This could be taking advantage of specific hardware, or maybe not handling use cases you don’t need. However, many available filesystems are already heavily optimized, so performance gains may be difficult to achieve.
If you must build your own filesystem, here are some things to consider.
Scope
Your system may not need all the features common in most filesystems. Things you might not need include:
- Folders
- Time stamps
- Permissions
- Variable file sizes
Limiting your filesystem to what you do need will reduce the complexity of your filesystem design, and therefore reduce development time and potentially improve performance.
One place to start when looking at features and interfaces you need is the POSIX standard, as many filesystems are POSIX compliant.
Metadata
Users of a filesystem mainly just think of a file as the data it contains. But additional data about the file must be stored. The size, name and location of the file are usually crucial data to store. Information about what space has been used and what is available for use is also a common piece of metadata.
When designing your metadata, there is choice of how much metadata you want. Storing more metadata may make it faster to parse/access the data at the cost of less disk space for actual data and more time spent updating the metadata.
Fragmentation
It would be convenient to keep files entirely contiguous, but often not possible. This leads to file fragmentation.
File fragmentation decreases performance as it prevents consecutive reads and adds additional metadata that must be stored for the file. Your design will therefore want to reduce fragmentation when possible.
Hardware
Filesystems must be suited to the hardware they are running on.
Especially in embedded, flash storage is quite common but presents additional complexity. For example, in most flash devices, after writing data to a memory page, new data cannot be written again to that page until a larger memory block is erased. Due to this mechanism, filesystems must often write the data elsewhere instead of rewriting data to the same location.
Additionally, characteristics of the hardware such as the page size in a flash device should be considered. How well the filesystem is aligned with the hardware can have a large impact on the performance of the filesystem.
Ready for launch?
A robust filesystem takes time and resources to build. If you haven’t budgeted for a filesystem while facing an aggressive schedule, you could also find yourself up against a wall. In contrast, consider your filesystem early on in the development process and you’ll reach your goals with greater confidence.
DornerWorks has years of experience building complex embedded systems that require every bit of space to be used efficiently. Whether it’s a finicky filesystem, or a completely new device built from the ground up, we can guide your next project to success. Contact us today and set up a free consultation.
Visit Our Site