Seeking ideas for MMD 4
MultiMarkdown v3 has been around for about 2.5 years, and represented a HUGE improvement over v2:
- written in C, allowing vastly improved performance and use on more systems (e.g. iOS)
- PEG grammar allowed for more robust parsing rules and accuracy
- direct output to HTML, LaTeX, OPML, and more without requiring the use of XSLT files
This was possible because it was built on the peg-markdown project that John MacFarlane created, which was a great project.
As I have worked more with the code base in other situations (i.e. as a library embedded in MultiMarkdown Composer 2.0, and some early work on an iPhone/iPad app), I am increasingly bumping up against limitations and flaws in the code (some mine, some inherent in peg-markdown).
The short version is that John’s code was never designed to be used as a library embedded in another application that didn’t quit after processing the code. It works wonderfully when processing a file into HTML and then quitting. It doesn’t work so well when it parses the same file 100s or 1000s of times (or more). The memory leaks begin to add up, eventually causing the parent app to stall if left running long enough. It also runs into problems on multithreaded machines — the library can’t be run more than once at a time.
I have patched some of the leaks, but I am not convinced it is possible to patch all of them given the way the code is written. And even if I do patch all the leaks, there are still a few other issues that will be virtually impossible to completely eliminate.
After thinking about this off and on for a few months, I think have come up with a way to rewrite the code so that it is cleaner, easier to maintain, and (hopefully) will not have any memory leaks. The new approach will also open up some flexibility to add new features and ways of using the code.
As I look into starting this, I want to get feedback for features that others (especially programmers) would like to see. For example, I could include the ability to export a parse tree of sorts that could be used in a syntax highlighter.
I AM NOT LOOKING FOR SUGGESTIONS TO THE MMD SYNTAX OR FEATURE SET. I AM LOOKING FOR FEATURES IN THE UNDERLYING ENGINE. DO NOT REQUEST A NEW SYNTAX FOR TABLES OR FOOTNOTES, ETC.
I have some interesting ideas already that may dramatically increase performance under certain situations and add the ability to do completely new things. Some may be achievable, some may not. I have also not decided how much will be part of the open-source MMD project, and which may be features that will be part of Composer (or other apps), but not the core MultiMarkdown code.
What is decided (or at least probably decided…):
The parser will still be based on a PEG. This PEG will basically be the same as peg-mmd/peg-markdown. It has proven to be quite reliable, accurate, and fast. The changes will be built around the c code, not the PEG. This should allow me to keep in sync with any improvements that John MacFarlane adds to peg-markdown.
The core of this work will become MMD v4, and will remain open-source. It will still be able to be compiled on just about any platform, and will be easier to include in other projects.
To most users the changes will be relatively invisible. The input and outputs will be the same, but the underlying engine will be dramatically different. I suspect performance will not be significantly better, and I hope it won’t be significantly worse
It will be easier to add (or change) the output routines (e.g. creating the resultant HTML, LaTeX, etc.)
If there’s anything you think should be added, now’s your chance!
Follow me on twitter
Subscribe to atom feed