Why do most log files use plain text rather than a binary format?

systemd famously stores its log files in binary format. The main issues I have heard with it are:

1. if the log gets corrupted it's hard to recover as it needs specialist tooling
2. you can't use standard tools such as vi, grep etc to analyse them

The main reason for using a binary format (to my knowledge) was that it was deemed easier for creating indices etc i.e. to treat it more like a database file.

I would argue that the disk space advantage is relatively small (and diminishing) in practice. If you want to store large amounts of logging then zipping rolled logs is really quite efficient.

On balance, the advantages of tooling and familiarity probably would err on the side of text logging in most cases.

There are a lot of debatable presumptions here.

Logging has been an integral part of (almost) every job I've had. It is essential if you want any sort of visibility on the health of your applications. I doubt that it is a "fringe" use; most organizations I've been involved with consider logs very important.

Storing logs as binary means you must decode them before you can read them. Text logs have the virtue of simplicity and ease of use. If you're contemplating the binary route, you might as well store logs in a database instead, where you can interrogate them and statistically analyze them.

SSD's are more reliable than HDD's nowadays, and the arguments against lots of writes are largely moot. If you're really worried about it, store your logs on an ordinary HDD.

Log files are a critical part of any serious application: if the logging in the app is any good, then they let you see which key events have happened and when; what errors have occurred; and general application health that goes beyond whatever monitoring has been designed in. It's common to hear about a problem, check the application's built-in diagnostics (pop open its web console or use a diagnostic tool like JMX), and then resort to checking the log files.

If you use a non-text format, then you are immediately faced with a hurdle: how do you read the binary logs? With the log-reading tool, which isn't on your production servers! Or it is, but oh dear, we've added a new field and this is the old reader. Didn't we test this? Yes, but nobody deployed it here. Meanwhile, your screen is starting to light up with users pinging you.

Or perhaps this isn't your app, but you are doing support and you think you know it's this other system, and WTF? the logs are in a binary format? Ok, start reading wiki pages, and where do you start? Now I've copied them across to my local machine, but - they're corrupted? Have I done some kind of non-binary transfer? Or is the log-reading tool messed up?

In short, text-reading tools are cross-platform and ubiquitous, and logs are often long-lived and sometimes need to be read in a hurry. If you invent a binary format, then you are cut off from a whole world of well-understood and easy-to-use tools. Serious loss of functionality just when you need it.

Most logging environments strike a compromise: keep the current logs readable and present, and compress the older ones. That means you get the benefit of the compression - more so, in fact, because a binary format wouldn't shrink the log messages. At the same time, you can use less and grep and so on.

So, what possible benefits might arise from using binary? A small amount of space efficiency - increasingly unimportant. Fewer (or smaller) writes? Well, maybe - actually, the number of writes will relate to the number of disk-commits, so if log-lines are significantly smaller than the disk blocksize, then an SSD would be assigning new blocks over and over anyway. So, binary is an appropriate choice if:

• you are writing huge amounts of structured data
• the logs have to be created particularly quickly
• you are unlikely to need to analyze them under "support conditions"

but this is sounding less like application logging; these are output files or activity records. Putting them in a file is probably only one step away from writing them to a database.

EDIT

I think there's a general confusion here between "program logs" (as per logging frameworks) vs "records" (as in access logs, login records etc). I suspect the question relates most closely to the latter, and in that case the issue is far less well-defined. It's perfectly acceptable for a message-record or activity log to be in a compact format, especially as it's likely to be well-defined and used for analysis rather than troubleshooting. Tools that do this include tcpdump and the Unix system monitor sar. Program logs on the other hand tend to be much more ad hoc.

Why do most log files use plain text rather than a binary format?

Search for the word "text" in the Unix philosophy Wikipedia article, for example you'll find statements like:

McIlroy, then head of the Bell Labs CSRC (Computing Sciences Research Center), and inventor of the Unix pipe,[9] summarized the Unix philosophy as follows:[10]

This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.

Or for example, from Basics of the Unix Philosophy,

Rule of Composition: Design programs to be connected with other programs.

It's hard to avoid programming overcomplicated monoliths if none of your programs can talk to each other.

Unix tradition strongly encourages writing programs that read and write simple, textual, stream-oriented, device-independent formats. Under classic Unix, as many programs as possible are written as simple filters, which take a simple text stream on input and process it into another simple text stream on output.

Despite popular mythology, this practice is favored not because Unix programmers hate graphical user interfaces. It's because if you don't write programs that accept and emit simple text streams, it's much more difficult to hook the programs together.

Text streams are to Unix tools as messages are to objects in an object-oriented setting. The simplicity of the text-stream interface enforces the encapsulation of the tools. More elaborate forms of inter-process communication, such as remote procedure calls, show a tendency to involve programs with each others' internals too much.

Anyone can make this for like two days in his spare time, why don't people do this?

Storing the log file in binary is only the beginning (and trivial). You'd then need to write tools to:

• Display the whole log file (edit)
• Display the end of the log, without reading the beginning of it (tail -f)
• Search for stuff in the file (grep)
• Filter to only display selected/interesting stuff (using an arbitrarily complicated filter expression)
• Email the log to someone else who doesn't have your log-file-decoder-software
• Copy-and-paste a fragment of the log file
• Read the log file while the program (which creates the log file) is still being developed and debugged
• Read log files from old versions of the software (which are deployed on customer sites and running).

Obviously software can and does use binary file formats too (e.g. for relational databases) but it's not worthwhile (in a YAGNI sense), usually not worth doing, for log files.

Log files are in text format because they can be easily read using any type of text editor or by displaying the contents via console command.

However, some log files are in binary format if there is a lot of data. For example, the product I am working on stores a maximum of 15000 records. In order to store the records in the least amount of room, they are stored in binary. However, a special application must be written to view the records or convert them to a format that can be used (e.g. spreadsheets).

In summary, not all log files are in textual format. Textual format has an advantage that custom tools are not needed to view the content. Where there is a lot of data, the file may be in binary format. The binary format will need a (custom) application to read the data and display in a human readable format. More data can be packed into a binary format. Whether to use textual format or binary format is a decision based on the amount of data and ease of viewing the contents.

In embedded systems where I might not have an output channel available during run-time, the application can't afford the speed hit imposed by the logging, or logging would alter or mask the effect I'm trying to record, I've often resorted to stuffing binary data into an array or a ring buffer, and either printf()ing it at the end of the test run or dumping it raw and writing an interpreter to print it as readable. Either way, I want to end up with readable data.

In systems with more resources, why invent schemes to optimize what doesn't need optimizing?

The two main questions you would want to ask before choosing between text and binary are:

• Who is my audience?
• What content do I need to convey?

A common opinion is that the audience of a log message is a human being. This is obviously not a perfect assumption, because there are plenty of log crawling scripts out there, but it is a common one. In this case, it makes sense to convey the information in a medium which humans are comfortable with. Text has a long standing tradition of being this medium.

As for content, consider that a binary log must have a well defined format. The format must be well defined enough for other people to write software which operates on those logs. Some logs are quite well structured (your question lists several). Other logs need the ability to convey content in a less-well-defined natural language form. Such natural language cases are a poor match for binary formats.

For the logs which could be well described in binary, you have to make a choice. Because text works for everyone, it is often seen as the default choice. If you log your results in text, people can work with your logs. It's been proven thousands of times. Binary files are trickier. As a result, it may be that developers output text simply because everyone knows what that's going to behave like.

Log files are intended to aid debugging of issues. Typically, hard drive space is much cheaper than engineering time. Log files use text because there are many tools for working with text (such as tail -f). Even HTTP uses plain-text (see also why don't we send binary around instead of text on http).

Additionally, it's cheaper to develop a plain-text logging system and verify that it works, easier to debug if it goes wrong, and easier to recover any useful information in case the system fails and corrupts part of the log.

We count on unit testing for attaining and maintaining the robustness of our software. (Most of our code runs in a server, headless; post-operation analysis of log files is a key strategy.). Nearly every class in our implementation does some logging. An important part of our unit testing is the use of 'mock' loggers that are used when unit testing. A unit test creates a mock logger and provides it to the item being tested. It then (when useful/appropriate) analyses what got logged (especially errors and warnings). Using a text-based log format makes this much easier for much the same reasons that analyses performed on 'real' logs: there are more tools at your disposal that are quick to use and adapt.