Blackboard’s Complexity Problems

Reactions to this year’s BbWorld are starting to roll in, and I’d like to add something to Michael’s recent post on Blackboard’s Messaging Problems. Yes, Blackboard could be better at communicating (can’t we all?), but I believe that Blackboard’s more challenging problems are rooted in the complexity of its software rather than the perplexity of its marketing messages. (Good work on Ultra though – that’s looking fantastic. If that’s what Ultra is. Which now I’m not sure what it is. But good work nonetheless. Though I haven’t come across anyone actually using it yet…)

Both confusing messaging and technical complexity present real problems, but while messaging can be (relatively) easily corrected, the technical complexity is a lot more difficult to untangle. Let’s examine.

Architectural Components

The most major complexity of Blackboard’s flagship Learn LMS is the sheer combination of supported configurations that it supports, even in its latest release. It can be run on different versions of Windows, or Linux, or Solaris. It can be run with databases including multiple versions of Oracle or SQL Server (PostgreSQL support was also rumored at one point, but I still don’t see it). Each of these can be run with different combinations of OS and database updates, hotfixes, and patch sets. The software runs on a version of Java which reached its end-of-life several months ago (and will no longer receive any updates, security or otherwise, else the Java version would be still another variable). It can run on virtualized environments or bare metal. It can be hosted by Blackboard or self-hosted. And these combinations don’t even consider the additional combinations that Michael’s post references – adding even more dependencies on software hosted or delivered by Amazon and though others.

The application that runs on top of all of these components is a well-intentioned but clunky amalgamation of many different pieces. It has flavors of the historical Blackboard and of the WebCT architects. There’s a little bit of ANGEL mixed in. Some of the design came from seasoned software architects with decades of experience while other bits came from media studies majors (especially one really smart one in particular). Some was built when the company was still a startup, others the artifacts of various rearchitecting efforts. Some pieces were jammed in or bolted on though various acquisitions.

Today, it’s not just the pieces that Blackboard itself owns that are important. In-line document rendering is provided by a third-party called Crocodoc. Parts of the video recording and embedding capabilities are provided by Google and YouTube, a fact that became painfully clear when all embedded course videos in all Blackboard environments across the world recently, apparently stopped working. Unlike companies like Instructure that can update every single one of their customers instantly and simultaneously when something breaks or D2L who only needs to focus on the Windows technology stack and is making steady progress towards periodic automatic updates for both hosted and self-hosted institutions, Blackboard still faces every combination of variables imaginable (and arguably still more after this year’s BbWorld).

To be fair, few other companies support this sheer raw complexity of configuration combinations, so kudos to them for holding it together for so long. In comparison, Microsoft supports an incredible array of Office product versions across Windows and Mac OSs and seems to do well compatibility-wise. Apple in contrast contains the number of combinations of OSs and devices it supports. I suppose WordPress is a worthy comparison in that it is hosted on any number of different combinations of OSs, databases, and PHP versions. But they seem to have it figured out, too, as the last WordPress update I ran took one click in a web UI and less than 10 seconds. Which leads me to my next point…

Modularity and Updates

Blackboard for many, many years has had this wonderful plugin framework called Building Blocks. It’s seriously cool (and I’m not just saying this because I spent a decade of my life building them). Building Blocks were “apps” 10 years before the concept became mainstream. And when Blackboard announced that they were (finally) modularizing the Learn product into their own Building Block plugin framework to make updates easier, I was really excited. The approach was rushed into use as a way to increase the pace and decrease the time to availability of bug fixes. This was a good goal, but it fast-tracked some thinking-through of important details. And as a result, when you had a bug, not only did the Blackboard Support team have to ask what version of Blackboard you were running, but also which specific versions of each plugin you were running.

Updating these plugins sometimes required restarting Blackboard services (ie: downtime); other updates did not.  Some updates had to be performed in a certain sequence, or else they would not work. Or one time I remember the official documentation specified an order, but it was the wrong one. Or once the specified order was technically impossible to achieve. There was also no real way to roll back once the install button was clicked. And so the risks to keeping the software up-to-date seemed to actually become even greater. Oh, and there were still releases/service packs and hot fixes to keep track of despite this welcome improvement. Sometimes those “upgrades” actually blew away the Building Block updates made between the release of the official updater/installer and the last batch of Building Block updates. And of course, there’s that pesky detail that every institution still gets to decide at its own pace and on its own timeline when to update.

APIs and Integration

Speaking of Building Blocks, there’s one last complexity to Blackboard’s software that can be observed indirectly through its APIs. You can comb through the latest API documents to make your own judgment calls, but from my technical vantage point, I see:

  • 6 representations of a “course”(AdminCourse, Course, CourseCourse, CourseSite, CourseVO, Organization)
  • 4 representations of “enrollment” (Enrollment, CourseMembership, CourseMembershipVO, StaffAssignment)
  • 3 or 4 representations of “user” (User, UserVO, UserInfo, Person)
  • 2 (public) representations of “grades” (anecdotally there are several others, which is why grades in their mobile app historically haven’t always matched the instructor gradebook which sometimes didn’t match the student view of the gradebook – and why it’s so hard to fix them all)
  • 10ish representations of “course content” (Content, ContentFile, ContentFolder, ContentVO, CSEntry, CSFile, BbFile, ChildFile, CourseDocument, LOItem (learning object))

Needless to say, it’s not only confusing for external third-party developers but likely to Blackboard’s own in-house developers which ones are the right ones, or best ones, or most-appropriate ones to use.

As was the case in 2013 when this blog started tracking LMS data, Blackboard still has the largest variance of installed versions of all the major flavors of LMSs (though to their credit, it is getting better). But the challenges are still far from over. I suspect foregoing the Spring 2015 release was one part of Blackboard’s strategy towards bringing all of their customers closer together and narrowing the version spread.

But my bottom line remains. Technical problems are much harder to solve than marketing ones, and the complexities we see here make every Blackboard installation unique in some nuanced way or another.  Staying this course is not sustainable.  If I were Blackboard, I’d focus on decreasing software complexity as an important goal rather than replaying the “rebundling the licenses” shell game. A good product may sell itself, but a complex one hastens its own morbidity.

-GK

7 thoughts on “Blackboard’s Complexity Problems

  1. Most of what you are describing here are typical and almost inevitable problems of a 15-year-old enterprise application. Getting rid of old APIs that customers may have built on heavily, breaking a lot of third-party integrations along the way? That’s hard. How do you tell customers that you will no longer support the database they are licensing, running on, and know how to use? That’s nearly impossible. The only way to clean up is to persuade customers to move to a newly architected cloud version that offers enough enticements that customers are willing to give up some of their customizations (and neuroses about self-hosting). Which is exactly what Blackboard is trying to do. (By the way Blackboard’s cloud platform runs on PostgreSQL.) Whether they will succeed or not is still an open question, but cleaning up 15 years of cruft and technical debt is not an easy problem to solve.

    And no, WordPress is not a fair comparison (much as I love it). It is a vastly simpler application. Apple and Microsoft control their entire ecosystems, so they’re not fair comparisons either. You want to look at enterprise apps for apples-to-apples. If you want a fair comparison, try Peoplesoft. If the very thought makes you shudder in horror, then you starting to get the scope and nature of the problem Blackboard faces. (Note: Oracle is trying to solve their Peoplesoft problem the same way Blackboard is trying to solve their Learn problem, i.e., by enticing people to a new cloud architecture.)

  2. I forgot to mention one other way in which the Peoplesoft comparison is apt: As of 2010, Peoplesoft was still running a significant amount of COBOL code. For all I know, they might still be running it today.

  3. “breaking a lot of third-party integrations along the way”
    >> Yes – this is exactly what I’m saying. Break them. But do it responsibly. Keeping APIs and other component technologies up-to-date is a solvable problem. Keeping crufty APIs around for another decade because 3 people might be using them demonstrates a very dated attitude that I would argue puts us all in a worse place. There are many acceptable approaches to responsible API deprecation, for example:

    • Google tends to phase out APIs over time and replace them, often giving more than a year’s notice at a time to switch.
    • Salesforce keeps multiple versions of APIs running in parallel for a period of time, but it always remains clear which one to use. The timeline is long enough so that if there are misses in new versions, there’s time to fix/address before the old one goes away.
    • Java provides a great example of forward compatibility. A lot of Java code written years and years ago still works fine – specifically because they’ve put effort into consciously architecting change.

    Speaking of Oracle/Java, everyone loves to hate on PeopleSoft. I do too. But it’s way, way more complicated than an LMS. And yet I’ve honestly not yet seen an example where things have gotten this “messy” with it. And when you file a support ticket, that stuff tends to actually get fixed right (caveat: all of this in my experience). As for the COBOL comment, it still works reliably, so why change it? I’m sure my bank still uses it, too. My observation is that Blackboard’s situation has been getting avoidably unwieldy not necessarily outdated.

    As for your comment on WordPress, I’m not sure what you’re using your LMS for, but to this day in my experience it’s never been used for anything more than a glorified file share with a gradebook, maybe a forum and a quiz tool thrown in for good measure. How LMS technologies are being used is (finally, thankfully) appearing to evolve, but as for what LMSs are capable of doing, sadly I’ve not seen much change in a while. Yes, I know, LTI yadda yadda. It’s cool and useful. But even Instructure, the new hotness, found that the vast majority of their new, online classes are centered around simply hosting basic content. Not all that different from WordPress, and I bet WordPress could survive the thousands of hits during finals week (after all it regularly survives many millions for these popular sites). I know that you see it as apples and oranges, but I still see it (today) more as clementines and tangerines (LMSs being whichever one has the annoying seeds).

    Software needs maintenance just like cars, houses, human bodies, roads and bridges. People don’t always see this, both literally and figuratively, because of its virtual nature. And sometimes, yes, vendors should tell universities to get their technology ducks in a row. This is what I consider being a leading vendor – having the desire to move everyone forward together not stuck together in the duct tape and baling wire of the past. I also think universities should not get a pass for being so dated. Frankly, outdated technologies that we’ve let linger far too long are a huge part of the problem. It’s not an impossible situation, though, and saying this only reinforces a negative, hopeless, helpless attitude. This has to change.

    Still, all of this mess though avoidable remains fully addressable. I think Blackboard has chosen its strategy with Ultra. But by putting it off for so long there is now a lot of major work to be done instead of a steady stream of less risky changes/improvements. I do wholeheartedly agree with you on this point though – if they can pull this off, the future does indeed look a lot brighter.

  4. Come on, George. You’re way more sophisticated than that. The test engine in any LMS alone probably has more lines of code in it than WordPress does. Grade books are ridiculously complicated. And then there are the permissions structures. It doesn’t matter how the average faculty member is using it; what matters is the code the vendor has to maintain.

    And you can’t complain about the version of Java Blackboard is on in one breath and say “COBOL is fine” in the next. What makes you think that Peoplesoft’s COBOL is more up-to-date and secure than Blackboard’s Java? Do you know how hard (and expensive) it is to even find a COBOL compiler these days?

    As for not having seen Peoplesoft breaking badly, maybe that’s because you’re not responsible for developing or maintaining it. I used to work on Peoplesoft when I was at Oracle and I talk to a lot of Peoplesoft customers as an analyst. Believe me, things break. Badly. Sometimes costing customers hundreds of thousands or even millions of dollars.

    Enterprise software ages poorly. Always. D2L, who you seem to be giving a free pass here, has some of the same problems. Why, for example, are they still struggling to extract data for analytics after spending a good three or four years (or more) working on the problem? I’ve seen Sakai struggle with aging problems. I’ve seen Moodle struggle with aging problems. And none of them have an installed base that’s anywhere close to Blackboard’s. It’s fine to hold the new guys accountable for making progress on all these accumulated problems, but you make it sound like they are uniquely bad and have little excuse for struggling with their challenges. In my view, neither of these things is true.

  5. Michael, sure – of course I’m oversimplifying things to make a point, but you’re vastly over-complicating what these systems need to be. I feel that your vast experience with crappy enterprise systems may be coloring your ability to see this and hesitate to hold this particular vendor’s feet to the fire because you know how the sausage gets made. From my perspective, the company made conscious choices, and this is where they ended up. I realize that what we see today is the result of a lot of choices adding up over a long period of time, but frankly that’s what this exec team signed up to fix. I don’t envy them, but I do expect a team of Blackboard’s caliber to have made more progress towards figuring this out by now and more clearly communicating the plan. Though enterprise software may have “aged poorly” in the past, saying this now is nothing more than an excuse for it to continue to suck well into the future. I may not be able to convince you, but it doesn’t always have to be this way. This is a very poor attitude and outlook to hold. I hope the angle that you’re coming from is instead understanding the undesirable circumstances of the past to proactively avoid them in the future.

    Since you brought the others up, D2L in contrast is actively executing on their plan to keep their customers updated to their current version and has been for over a year. They have had hiccups (I’m one of them), but unlike Blackboard, a growing percentage of their client base is actually using their newest version … because it’s actually real. Moodle had struggles getting over the 1.9->2.0 hurdle but seems back on track. The cloud progress they’ve made is impressive. Sakai is making more progress, more quickly, than I’ve seen in years. Like your recent post, I would have expected Instructure to have faltered in some medium-severity, highly-visible way, but they haven’t – because IMO they’ve paid attention to the details. Analytics IMO is a separate mess that no one’s really figured out yet, so I give everyone a pass there. Personally, I don’t think that it belongs as part of the LMS architecture anyhow but rather somewhere separate. There are other areas of a university where sophisticated analytics are much more useful than inside an LMS right now, and I’m not convinced that the analytics capabilities within LMSs are mature enough to do much more beyond identifying lagging students who any good instructor would know about anyhow – it just makes it easier and more apparent to the rest. Don’t get me wrong, there’s some great analytics stuff going on out there. It’s just not happening predominantly inside the LMS.

    As for the gradebook, I do intend to be flippant when I say Excel can do everything I’d want my LMS gradebook to do – and it’s probably more likely to give me the correct grade calculations (I can personally attest to this last point being true under certain conditions in certain LMSs). Excel’s been around since … the 1980s? Calculating a column or row of values is a solved problem. Why the unnecessary complexity? Probably because some poor product manager couldn’t say no because some sales VP needed to win an RFP. This is what happens when software execs who are not familiar with the implications of technical complexity and technical debt don’t factor it into both strategic and operational decisions.

    As for PeopleSoft, I actually work with it every day (these aren’t just LMS-related things either). Again, the gripes (from the development team that I work on) are many, but the system works, and upgrades going back at least 2 years since I’ve been there have been consistently successful – a lot of frustrating work, but successful. I’m sorry that things didn’t work out for you, but they generally are working for us. I’m not sure what your point was there, exactly, but I think you helped to reinforce my point that complex systems are bad for business. For what it’s worth, we’re stripping a lot of complexity out of our many PeopleSoft integrations/customizations to make them simpler, to update them to be more standards-compliant, and to take advantage of PeopleSoft’s latest supported integration mechanisms.

    Cleansing fire is a good thing, and it is for this reason that instead of maintaining things that don’t work well for decades on end and layering more crap on top of an unstable base, I lean more in favor of becoming comfortable with revisiting and enhancing things on a more frequent, recurring basis. At the end of the day – people just want systems that work and do what was promised of them. I hope that Blackboard can keep the promises it’s made over the past several weeks, but that does not change the nature of the organization’s current, complex reality.

  6. I guess we’ll have to agree to disagree on this one. I don’t see myself (or Phil) as giving Blackboard a free pass. In our view, understanding how the sausage making is done is an asset, not an impediment. Trying to understand what a big company is doing to transform itself and modernize decades-old software is a little bit like trying to understand what’s happening in the ocean by looking at the surface of the water. You can infer some things, but you’re blind to a huge amount of it. When there’s nothing going on on the surface, there may be nothing going on below or a lot going on below. As analysts, part of our job is to make some inferences about what might be going on below and how that might eventually make itself known on the surface. Phil and I both see signs that Blackboard could be a contender for market share against Instructure again if they are able to move Ultra forward at a reasonable pace and get their messaging in line in the next twelve months. (We do not see signs that D2L will be a contender for market share in the same time period, by the way.) We are not giving Blackboard a passing grade, but neither are we giving them a failing grade. Based on our knowledge of the sausage-making—both technical and organizational—we are giving them an “incomplete” at the moment.

    The ultimate test will be market share numbers like the ones you publish here at edutechnica. If Blackboard does get its act together in the next 12 months, then we should see some directional changes in the next 24 months. We think it’s possible (but by no means inevitable) that they will, based on what we are seeing. People who read us to help them evaluate their LMS choices want to know not only how the company is doing today but how it is likely to be doing two or three years from now. So we take the long view.

  7. It’s not just Blackboard’s LMS products that are suffering. from technical debt. Let’s not forget they acquired both Edline and SchoolWires, companies that provide a proprietary CMS’s for powering district and school web sites.

    This is taken directly from an RFP for a large school district looking for a better solution.

    “[school district] began an aborted upgrade to the Edline platform, including design of sites and migration of content in 2015. This process was abandoned when Edline was replaced with the newly acquired and markedly different Schoolwires platform and its associated communication, content management system (CMS), mobile application and, social media tools. As we reconsider our options via this RFP, [school district] desires the capability to integrate all of these functionalities with one consolidated service contract.”

    Blackboard is trying to build a large K12 vertical. I get it. It’s just business, but when your private equity owners care more about putting themselves on the market for $3.5B than actually investing the time and money into getting their software platforms (including the web site CMS’s) modernized, then I fear that Blackboard will never figure it out.

    School districts are fed up. They are tired of spending money year after year and getting shortchanged on quality.

    Personally, I believe the solution is in open source software. More and more school districts are abandoning their contracts with proprietary technology vendors and going with open source solutions like Drupal and WordPress.

Comments are closed.