Literate programming vs reasonably documenting your code

Neither

I have a frame challenge for you, you write:

"Could it be that literate programming was a solution to problem that was tackled meanwhile through other means?"

I posit that this problem hasn't been tackled at all, at least not in practice. All the "solutions" here work in theory, and maybe in settings where everyone from the CEO to the junior programmer is convinced of it's use, skilled enough to execute it and has the discipline to never stray from the path.

The only practical example of this I'm aware of (though I'm sure there are more) is the software for the space shuttle

In my personal experience, even the best documented code has a lot of open questions and places where the documentation diverges from the code (no matter commented, literate, wiki-ed or otherwise documented).

The most successful projects I've encountered however would, bar one, not even document their code at all but in stead focus on good naming, good structure, code reviews and taking time to get new devs up to speed.

So in my case, the answer to the question in the title would be: Neither.

tl;dr - README.md is the modern day heir to literate programming

First of all, Knuth invented literate programming because he needed it for typesetting his books digitally. This was around 1980, making it probably the oldest software package in common use today (not counting mainframes).

As he wanted to teach a subject, elaborate explanation of the actual code was paramount. You most likely do not need this today. Also a lot of the features provided (because the assembly language he used - Standard Pascal - didn't) are now implemented in the languages themselves.

What do we need?

• Documentation close to the code.
• Documentation contains live code, not a dead copy.
• Documentation in a form that is a first class citizen on the web (i.e. usable with a browser)
• Documentation written in a language that is version control friendly, that is plain text.

GitHub is probably the main stream provider of what we have today which in practical use is the direct successor of literate programming, namely README.md files which are written in the Markdown language, and rendered when navigating the source (this is the really important bit). This allows you to easily document and describe your program, and Markdown is easy to learn. The ability to have the Git repository be both code and documentation is a very important milestone!

I did an experiment to see if I could explain how my "Hello, World!" in Dagger 2 (a Java dependency injection framework) was put together at https://github.com/ravn/dagger2-hello-world as a single file being both Java and Markdown (in the spirit of literate programming) and it came out pretty well. I then learned that the AsciiDoc language can refer to snippets in other files (to get live code in the documentation), but I have not yet tested it fully.

Literate programming is great in situations where the code is mostly there in support of the prose. That's why Jupyter notebooks and similar are common for scientific programming. I also use it when I am teaching a programming workshop.

In other situations, people often mistakenly think of the comments as being for humans and the code as being for the computer. If that were so, we may as well write in machine code, because the computer doesn't care. Instead we write in high-level programming languages because it's easier for humans to read and write.

Maintainability isn't achieved by "saving" code with copious comments. Messy code is really difficult to write clean documentation for, especially if you're asking the same person to write both. Clean code mostly stands on its own, with comments and other documentation playing a supporting role.

Wikipedia says:

"The literate programming paradigm, as conceived by Knuth, represents a move away from writing computer programs in the manner and order imposed by the computer, and instead enables programmers to develop programs in the order demanded by the logic and flow of their thoughts."

I don't think modern programming languages impose many order constraints that matter, so I don't see any big difference from using appropriate comments in the source code.

Also per wikipedia:

"The main intention behind this approach was to treat a program as literature understandable to human beings."

That seems a good aspiration. Good coding style ( appropriate choices of names, etc. ), combined with extra explanation in comments where appropriate is the answer, but it's mostly just hard work. I don't think there are any silver bullets here.

The modern attempts of the core idea of literate programming seems to be Jupyter notebooks.

In Knuth's own words: "The main idea [of literate programming] is to treat a program as a piece of literature, addressed to human beings rather than to a computer". That's pretty much what a Jupyter notebook is, a literature for human to read and share ideas, that just happens to contain interactive, executable code.

Generally, it seems like literate programming generally only makes sense if you're writing an academic paper or an article, and you want to include executable code in that paper/article.

literate programming was a solution to problem that was tackled meanwhile through other means

Yes, modern programming tend towards the thinking to improve readability of the code rather than adding comments. If code isn't readable without comments, then the code should be refactored so it is readable without comments. This is mainly through judicious use of intent revealing name and structures.

Literate programming is the nice idea that you can write your code together with an explanation or walkthrough of that code. Importantly, you are not constrained by the syntax of the underlying programming language but can structure your literate program in any way to want. (Literate programming involves chunks of code embedded into text, not comments into code.)

There are three huge problems with literate programming: it takes a lot of effort, there is little tooling, and changes become more difficult.

• Documentation always requires effort. Literate programming requires less effort than maintaining separate documentation of comparable quality. However, this amount of effort is still unwarranted for most kinds of code. A lot of code is not interesting and requires little discussion, it's mostly just delegating stuff to some framework. The kind of tricky logic that benefits from literate programming is comparatively rare.

• While there are various tools for literate programming (including Knuth's original WEB, and decent support in the Haskell ecosystem), they all suck. The next-best thing I've come across is org-mode, but that requires the use of Emacs. The problem is that programming is more than typing letters, it's also debugging and navigating code, which benefits greatly from an IDE-style experience. Auto-complete is non-negotiable! Literate programming tools also tend to require non-standard build processes, or mess up line numbers in error messages – not acceptable. If a tool makes your code easier to understand but harder to debug, that's not necessarily a good choice.

• Related to this is the issue that changes to literately programmed software become more difficult. When you refactor code, you also have to restructure the document. But while you have a compiler or linter to ensure that your code continues to make sense, there's no guarantee that you haven't disrupted the structure of the document. Literate programming is writing and programming to equal parts.

So while full-blown literate programming does not seem to have a place in modern software development, it is still possible to reap some of the benefits. Consider in particular that literate programming is now over 35 years old, so a lot has happened in the meanwhile.

• Extracting a function with a useful name has many of the same benefits of a chunk of code in literate programming. It's arguably even better because variable names get their separate scope. Most programming languages allow functions to be defined in an arbitrary order, which also allows you to structure the source code within a file in a sensible manner.

• Literate programming can be used to describe the “why” of a code in a human-readable manner. A somewhat related idea is to express requirements for your program in a both human- and machine-readable format, e.g. as suggested by BDD. This forms a kind of executable specification.

• Some markup languages have the ability to pull code snippets from your source code. This lets the code be code and lets you construct a narrative around these snippets, without having to duplicate, copy, or update the code. Unfortunately, the popular Markdown has no built-in mechanism for that (but RST, AsciiDoc, and Latex+listings do). This is possibly the best current alternative for creating literate programming-style documents.

What Knuth’s literate programming tools would allow you to do: Say you want a new feature. And for that feature, you need to create classes X and Y, and make changes to method in classes A, B and C. “Literate programming” would let you put all that in one source file, instead of say C++ where you had to add two header files and two source files for the classes, plus make changes in 3 different files.

This was very nice, but worked in Pascal (I think) only and I haven’t seen it implemented anywhere else.

Newer languages are getting closer. For example Java and Swift where you don’t have separate header and source files (Swift can extract the interface = what programmers need, not what the compiler needs like C++, don’t know what Java has). That’s a big step.

Other newer features are closures that pack up small bits of code that could be plugged into other classes. So the new classes you added for feature X might add bits of code to classes A, B and C like in Literate Programming, through language features and through having classes prepared for this. Not quite the same, but getting closer.

You still need documentation in literate programming, so this isn’t either / or.

Literate programming was a great idea at that time (and in fact I used it back then to write the only piece of software with a public release and some user-base, which never got any error report.)

But: there are some "buts":

• IDEs nowadays often use some kind of real-time-while-you-type compilation to show errors immediately. This probably would not work.
• Software used to be much smaller in the 80s. The complete codebase of TeX would be less than a single module in any piece of enterprise software today. Therefore, it is very difficult to structure software as a single train of thought now, which is where literate programming really shines.
• I am not aware of any source-level-debugger for literate programming. (Enlighten me via a comment if there is one.) This is a tool which I'd never go without.