D. Richard Hipp's GitHub Trip Report, Hacker News

I, the founder of Fossil, was invited and attended the first meeting of the GitHub Open Source Advisory Board in San Francisco on 2019 – 11 – 12. This post is a report of my adventures.

The only skeptic in the room

GitHub was a very gracious host, providing lunch before and dinner after The 5-hour meeting.

There were 24 people in the room, 5 or 6 of whom were GitHub employees. To nobody’s surprise, I was the only Git skeptic present. But everyone was very cordial and welcoming. It was a productive meeting.

It is not about Git

The GitHub employees seem to have all read at least some of my writings concerning Git and rebase and comparing Git and Fossil and so they knew where I stand on those issues. They are not in 100% agreement with me, but they also seem to understand and appreciate my point of view. They all seemed to agree that Git has UX issues and that it is not the optimal Distributed Version Control System (DVCS) design. Their argument is that none of that matters, at least not any more. Git is used by nearly everyone and is the defacto standard, and so that is what they use. Though I disagree, it is still a reasonable argument.

I offered a number of suggestions on how Git might be improved (for example, by adding the ability to check-in revisions to comments on prior check-ins) and their response was mostly “We don’t control Git”. I replied that GitHub designed and implement the v2-http sync protocol used by Git which has since become the only sync protocol that most Git user will ever experience and which significantly improved the usability of Git. I encouraged them to continue to try to push Git to improve. I’m not sure how persuasive my arguments were though.

Because: It isn’t really about Git anymore. GitHub used to be a Git repository hosting company. But now, they are about providing other software development and management infrastructure that augments the version control. They might as well change their name to Hub at this point.

For example, the main topics of discussion at this meeting where:

1. Issue tracking and tickets
2. Code review
3. Automated Continuous Integration
4. Maintainer workflow and triage

The GitHub staff says that the four pillars of their organization are

1. DevOps

Security

2. Collaboration

Insights

You will notice that version control is not on either of those lists. There has been a culture shift at GitHub. They are now all about the tooling that supports Git and not Git itself.

On the other hand, GitHub showed no hint of any desire to support alternative version control systems like Mercurial or Fossil. They openly assume that Open Source (mostly) runs on Git. It is just that the version control is no longer their focus. They have moved on to providing other infrastructure in support of Open Source.

Other Take-Aways

Documentation Is The Best Way To Say “No”

One of the most important and also the hardest jobs of a project maintainer is saying “no” to enhancement requests. If you try to take on every requested enhancement, your project will quickly loss focus and become too unwieldy to maintain. Participant “Paris” (whose full contact information I was unable to obtain) says: “Documentation is the best way to say ‘no’.” In other words, it is important to document why a project does things the way it does, as this will tend to prevent enhancement requests that cause the project to diverge from its original intent. I have also found that writing about the “why” tends to help one focus on the real purpose of the project as well.

Separately it was observed that documentation is the single most important factor in open-source adaption. The better your documentation, the more likely people are to use and contribute to your project.

Implications for Fossil:

1. I’d like to add the ability to include    PICandEQNmarkup in the middle    of both Markdown and Fossil-Wiki documents.
2. I think this Forum feature has been very successful, but there    are still many opportunities to improve the UX.

Additional ideas on how Fossil might be enhanced to support better project documentation are welcomed.

“The Tools Make The Rules”

This quote (from Tom Kremenek of the Swift project) is saying that your workflow will be defined and constrained by the tools you use. If you use inferior tools your productivity will suffer.

This is the argument I’ve made for years about the importance of being about to amend check-in comments and to view the descendants of check-ins. Git partisans offer all kinds of excuses about how their workflow does not need that. I tend to counter with “I never needed bisect until I had the capability. “The point is that once you have the ability to amend check-in comments and view the descendants of a check-in, your workflow becomes more efficient and you wonder how you ever survived without those capabilities.

What limitations or restrictions in Fossil are limiting productivity? Some possible ideas:

1. Poor tracking of files across renames
2. Inability to do shallow clones

Poor ticket UX

3. Inability to subscribe to notifications for changes to specific     tickets or files.
4. Difficulty implementing actions that are triggered by new     check-ins and / or new tickets. This is needed for Continuous     Integration (CI). Enhancements to thebackoffice    feature of Fossil would be welcomed.

General agreement that rebasing is not a good thing

Even among Git partisans, there seems to be a general agreement that rebase ought to be avoided. TheRebase Considered Harmfuldocument is not especially controversial.

An interesting insight from the LLVM developers: They use rebase extensively. But the reason is that downstream tooling controlled by third-parties and which was developed back when LLVM was still using SVN expects a linear sequence of changes. The LLVM developers would like to do more with branching, but that would break the downstream tools over which they have no control, and so they are stuck with having to rebase everything. Thus rebase supports historical compatibility of legacy tooling. It’s a reasonable argument in support of rebase, not one that I necessarily agree with, but one that I understand.

Summary And Concluding ThoughtsGitHub is changing. It is important to understand that GitHub is moving away from being a Git Repository hosting company and towards a company that provides an entire ecosystem for software development. Git is still at the core, but the focus is no longer on Git.

Fossil already provides many of the features that GitHub wraps around Git. Prior to the meeting, someone told me that “GitHub is what make Git usable. “Fossil has a lot of integrated capabilities that make GitHub unnecessary. Even so, there is always room for improvement and Fossil should be adapting and integrating ideas from other software development systems.

One GitHub-er asked me: “What would it take to get SQLite to move off of Fossil and on to Git. “Just to be clear to everyone reading this: That will never happen. Fossil was specifically designed to support SQLite development and does so remarkably well. Fossil fills a different niche than does Git / GitHub. Fossil will continue to be supported and enhanced by me (as well as others) well into the Foreseeable future.

Prior discussion of PIC markup in Fossil documentation was at postdb 217184 de. The EQN markup would work much the same way.

Since posting that a couple of months ago, I did actually try to write a PIC language interpreter in Javascript. But I discovered that Javascript is not a particularly good language with which to write a parser / compiler – or at least my limited knowledge of Javascript was insufficient for the task. Input from readers with better Javascript Skills would be appreciated here.

I have heard that it is now possible to compile C code into WASM. In particular, I hear that people are compiling “sqlite3.c” into WASM and running it in browsers. Perhaps an alternative approach would be to write a compact PIC translator in C (using Lemon to implement the parser) then compile the result into WASM for download to the browser. But I don’t understand how all of that works. If anybody has any insights and can help guide me through the process, that would be appreciated.

A question about lemon. IIRC it is a LALR parser generator similar to Bison.

Is the generated parser able to execute semantic actions as the AST is built? This and the ability to reach back to the lexer to control its execution will be necessary for PIC.

The reason for that is PIC’s macro system, which it has in lieu of proper procedures / functions. Macros can recursively call each other, and to not have this end in tears of infinite recursion this expansion has to be / is curtailed in code not executed, i.e. the untaken branches of if / else conditionals.

For the implementation of the language this means that semantic actions have to be executed as the input is lexed and parsed, and then reach back to the lexer with the outcomes of conditions to selectively skip / suppress macro expansion.

See pages 6-8 of thedpic docs.

With that in mind I wonder if it might be suitable to reuse an existing parser likedpic, instead of writing a new PIC parser, for compilation to WASM .

it is now possible to compile C code into WASM

Yes, and more to the point, WebAssembly support isapproaching universal adoption. It won’t run in ELinks, but this is a non-core feature that those with such niche needs can ignore.

I do wonder if server-side rendering to a virtual SVG would be a better plan, though.

Is there a place (someone’s repo on ChiselApp?) Where we could collaborate on the PIC renderer?

I seem to have created a Lemon grammar for PIC, but I cannot test it on real inputs yet, because first I’ll have to hack the lexer to understanddefines and substitute macros. (According to Kernighan and Raymond, they have to be done outside the parser, by pure sting substitution.)

Also, I haven’t started the work on the “virtual machine” to interpret the AST and output SVG.

The only skeptic in the room

Butyou are not alone!

You will notice that version control is not on either of those lists.

That’s … kind of mind-blowing. Perhaps they consider it an implicit entry in the list, since it’s a level of infrastructure which underpins every aspect of what they do. A “can’t see the forest for the trees” kind of thing.

What limitations or restrictions in Fossil are limiting productivity?

My own personal entry in that list is not new: the lack of a library interface. Obviously (for those who don’t know this tidbit of history) i intendedto tackle that with libfossil, and was well on the way to feature-parity with the fossil binary when chronic RSI (specifically, C8 nerve inflammation / damage in both arms) effectively kicked me out of the game in late 2014 (even now, my typing capacity is still reduced to about 10 – 15% of what it previously was, and i am still in forced early retirement due to the years-long duration of my medical leave).

Coincidentally enough, since yesterday i’ve been milling over a reboot of that work, focusing initially on features which would make it easier / possible to create customread-onlyclients, ignoring the ability to update / merge / etc. for the time being, simply to try to keep the scope in check. The motivation for that came from user elechak, whoposted a query based on the (mlink) table. Being able to plug that type of feature one’s own app / page / whatever would empower all sorts of apps we don’t / can’t currently have. Obviously once we have a library, the gateway is opened for any number of clients, e.g. tooling for reports, CI, and IDEs.

That said … my hands cannot reasonably commit (as it were) to that scope of project for the foreseeable future :(. Getting the library up and running again, now that fossil supports multiple hash types (libfossil predates those changes), would require an as-yet-unknown amount of effort.

Thatsaid, i don’t think it would take too awful much to get the read-only functionality working again. So … i’ll take this opportunity to bang the drum again and call for “seriously interested” coders toget in touchand help make that happen. (Forewarning: it wouldn’t be a small amount of code.) While i cannot personally commit to more than “casual” amounts of coding, my capacity for technical support is far less bounded and my capacity for moral support is practically limitless;) .

One GitHub-er asked me: “What would it take to get SQLite to move off of Fossil and on to Git.”

LOL!i haven’t laughed this hard since you explained the logic behind you getting taller with age!

i intended to tackle that with libfossil

Let’s not make the same mistake twice:libfossilfell into disrepair because it’s a second implementation of the formats and protocols within an ecosystem that’s not able to progress as fast as it wants to on a single implementation. (Thus this thread.)

Rather than resurrectlibfossil, I’d rather see that work went into refactoring the internals of Fossil to extract a library, divorcing the CGI and CLI interfaces from a core that only deals with the repo formats.

Some of this is already done well enough that all that’s needed is to draw the lines more explicitly, but I want to get to a world where the build process results in a staticlibfossil2.athat links to the Fossil binary, which can optionally be built as a shared object and linked into an app without linking tolibnss,libssl, etc. because the UI is a native GUI tool that only works on local repos and thus doesn’t need network clone and sync support.

The resulting library should be modular, so that if the third-party app also wants clone and sync support, it can include that as well.

Or, Plan C: expand the current web APIs to the point that one could write a full-function Fossil library that did nothing other than make HTTP API calls to a running Fossil instance.

Proposed Forum Enhancement

I would like to start a new Forum thread named something like “Libfossil Reboot” based on the previous comment. The new “Libfossil Reboot” thread would link back to Warren’s post, and Warren’s post should link forward to the new thread. But it would be a completely new thread, in as much as it is a tangent off of the original.

In other words, I’d like to create a simple mechanism to preventthread hijacking.

Somebody with good UX sense, please suggest a suitable interface for this new feature, or if the new idea makes no sense from a UX standpoint, please talk me out of it.

At risk of creating a hijack of a hijack (🤷‍♂️) what if a Moderator or higher level user could mark a post as being a topic fork, which then prompts the user for a new thread title. The existing post becomes the starting post of the new thread under the new title, and unlike current first-post-in-thread, it has an “in reply to” in its header pointing back to the other thread.

For normal users, all they see is that the Reply button changes to Moved or similar. Clicking it takes them to the start of the other thread, where they can now click Reply.

Don’t send the user straight to a reply: they need to see the new context first. They might then choose to reply further down-thread in the child topic.

(Edit: sorry, Richard – i didn’t see your note about the threadjacking until posting this: /. Further discussion will take place in the new thread if / when it starts.)

Fair enough.

Rather than resurrect libfossil, I’d rather see that work went into refactoring the internals of Fossil to extract a library, divorcing the CGI and CLI interfaces from a core that only deals with the repo formats.

That’s more or less originally what the goal was. There are, however, fundamental design decisions in fossil which make “bending it” into a library unfeasible. First and foremost is its allocation failure policy: simply die. Thatdrasticallyreduces the amount of code needed (as can be seen when comparing the same algos in fossil core to libfossil – a huge number of lines of code in libfossil are checking / handling result codes and allocation results, where fossil would simply die in most such cases). A library cannot feasibly have a die-on-OOM policy. (That is to say: i wouldn’t personally use a library which i knew would outright kill my app if the library encountered an error code or couldn’t allocate memory.)

That said: i would beCompletely on-board with such an effort. As Richard put it when it was proposed it in 2011, though, it would require “a Herculean effort.” (Though dictionaries / style guides differ on whetherHerculeanis capitalized or not when used that way, it would certainly have a massive capital H in our context!)

Or, Plan C: expand the current web APIs to the point that one could write a full-function Fossil library that did nothing other than make HTTP API calls to a running Fossil instance.

That’s kinda / sorta what the JSON bits do, and an HTTP-based API isEssentiallya link-at-call-time DLL. However, a JSON-based API would be a particularly poor choice for certain functionality, most notably transferring binary files of arbitrary sizes, as doing so requires encoding into a form JSON can use (huge strings or massive arrays of integers (byte values), both of which are larger than the binaries themselves).

Understand, I don’t mean that your intent in creating the library was a mistake, only that we now know that it clearly didn’t attract a big enough user community that you had to fend off the tendency to fork the libary when you had to give up on it, because you had multiple people all wanting to continue it. Knowing this, we shouldn’t repeat the decision.

Contrast Fossil: I can see this project forking if something drastic happened to drh or to the project direction. We’ve got enough people actively interested in working on it that wewouldhave to fight off a tendency to create multiple forks and not just one new successor project. I point that out not to show a danger to Fossil, but to show that it has abus factorabove 1.

Years now afterlibfossilstopped moving, I think we can say that it had a bus factor of exactly 1.: (

Therefore, efforts going forward should make Fossil itself more powerful, not divide precious resources.

As to the matter of difficulty, that’s why I proposed my previously-unnamed Plan B as a refactoring effort, rather than reimplementation. It’s something we can simply choose to strive toward, one bite at a time.

On the efficiency of a JSON API, many HTTP APIs switch content types to match the needs of the call. JSON returns may be the default, but binary and HTML returns are also legal based on context. Some calls may even have multiple forms. In that sense, Fossil UI is the HTML API to this library already, and/ rawis the binary interface. All I’m proposing is wrapping this in a way that you can use inside, say, a Fossil plugin for a programmer’s text editor.

Git is used by nearly everyone and is the defacto standard, and so that is what they use.

That’s pretty much the “Windows” argument. Windows hasn’t gone away in the face of smartphones, Chromebooks, and the cloud, but it has been pushed to the margins wherever practical. It’s probably a large part of the reason that the Windows phone and tablet efforts didn’t go anywhere.

It’s the point of view of an entrenched victor who becomes blind to the fact that theyarenow entrenched, and consequently can no longer move.

I’m not even sure this is a conscious choice. Perhaps a better metaphor is Gulliver strapped down by thousands of Lilliputians using ropes woven from process, standardization, and interdependency. Gulliver could choose to rise, but he’d destroy all the ropes in doing so.

1. Poor tracking of files across renames

That’s not the number one issue, just the most recent request. 🙂

Fixing this would be a nice quality of life improvement for those of us already using Fossil, but I doubt it factors very high in anyone’s decision about whether to use Fossil, or whether to leave Fossil for something else.

2. Inability to do shallow clones

That goes to scalability, and it probably does have a lot to do with Fossil’s adoption curve.

It is true that most software projects aren’t Linux-scale in terms of SLOC and complexity of development and thus don’t benefit from elements of Git’s design that serve the rare characteristics of the Linux kernel development project.

Nevertheless, it is also true that there are a lot of software projects bigger and more complex than SQLite yet still much smaller than Linux and thus could benefit if Fossil supported large-scale projects better.

3. Poor ticket UX

Now that we have Markdown in tickets (!!) my only serious complaint about it is the two-step filing process.

I see the value of the simplified filing interface for normal users, but when I’m logged in as a Setup user, it’s annoying to not have all of the ticket fields available to me to fill out at the same time as the initial filing.

This could be controlled by a user cap saying whether that user or category of users sees the full ticket UI on filing.

Speaking of user caps, another thing I anticipate is support for various SSO technologies: LDAP, OpenID, Oauth, SAML, etc. That in turn would require some way to map groups in those systems to user categories in Fossil, which would probably have to expand beyond the current hard-coded set of four.

4. Inability to subscribe to notifications for changes to specific tickets or files.

That sounds like it’d be pretty easy to implement. All it wants is someone to scratch the itch.

Not me; don’t care. 🙂

5. Difficulty implementing actions that are triggered by new check-ins and / or new tickets.

Server-side Tcl hooks?

I think the hardest part is making such hooks enforcing in the face of distributed clones. If I check something into my local repo but a push to the parent repo fails because of a Tcl hook refusing the new material, what then?

LLVM … downstream tools … stuck with having to rebase everything.

Wouldn’t a release branch (as opposed to release and version tags) solve this? Downstream projects could point at the branch they care about, which would get only a linear sequence of changes, pushed there by a release manager.

You can implement development, feature freeze, slush, and code freeze branches in this way within Fossil today, progressively restricting changes until a frozen blob of code lands on the release branch.

The only thing that might be nice in this regard is locking branches to particular users. Fossil’s DVCS nature makes absolute locking impossible, but part of such organized thinking is an assumption that the users aren’t actively malicious. If low-level Dev A checks something into the feature freeze branch directly, Fossil already has tools to fix that, and beyond that, it’s a social / admin issue, not a tech issue.

GitHub is moving away from being a Git Repository hosting company

They’re just moving up-stack, which is necessary to avoid becoming another commodity provider of Git service. If they’d continued to focus on Git, they’d be in direct competition with Digital Ocean and/ usr / bin / git. Theyhaveto move up-stack to continue propping up those Silicon Valley salaries.

Fossil doesn’t have the same need. It merely needs to remain useful to its users. I think this discussion should remain focused on that and possibly on attracting new users. But Fossil doesn’t have the same business needs as GitHub, Inc., a Subsidiary of Microsoft Corporation.