Internet of Silos

October 22nd, 2014

My friend Phil sent me a link to a Gartner article listing the “Top 10 Strategic Technology Trends for 2015” and I wasn’t surprised to see the “Internet of Things” at number two.

Now that mobile computing has become ubiquitous, there is a rush to network connect everything. Plus, there is a lot of money in it – take a look at the Nest acquisition. I’m not really certain this is a Good Thing™. My experience with low end network devices shows that corners that can be cut often are (i.e. crappy SNMP agents) and my guess is that in this rush to get things out the door we will end up with some serious issues.

One thing that can circumvent this is open source. By making the code transparent, especially for medical devices, there is a strong chance that major issues could be identified and corrected.

If, and that is a big “if”, we could get people to standardize around various open source software for the IoT there is a chance we can prevent the “Internet of Silos”. I first heard that term in a BBC article where it was announced that ARM (a British company) was creating an open source programming language for devices powered by its chips.

As you might expect, the article was short on specifics, but if the language is any good and the license is permissive, perhaps other chip manufacturers will port it. At a minimum it should encourage companies like Intel to also open source the technology used for their chips and the community might be able to build wrappers around both of them.

No matter what happens, we hope to support any management capabilities they introduce. We are actively working on making OpenNMS almost infinitely scalable to be able to handle the needs of the IoT, from insanely fast data storage (Newts) to highly distributed polling and data collection (Minion). We hope that the open nature of the platform will encourage more and more product vendors to use OpenNMS for their element management system, and then at least on the management front we can prevent the silos.

Conferences? We Got Yer Conferences

October 19th, 2014

This is my “oh” face

as in “Oh how awesome is All Things Open”.

Last year was the first edition of the All Things Open conference in Raleigh, NC. I was very happy that we could be a sponsor, and the above picture was taken during my talk.

We ended up with about 50% more people than were expected, and the keynotes were standing room only. It was really cool to see such a turnout, especially since it sort of validates the Raleigh area as a center for open source excellence.

This year we will have a booth where you can come by, get some OpenNMS swag, and hear about the pending release of OpenNMS 14 (yes, fourteen) which is only a few days away.

Now, “oh” could also mean “oh-hi-oh” as in the Ohio LinuxFest. Directly after All Things Open, the Ohio LinuxFest will be held in downtown Columbus this weekend. This has been one of my favorite open source conferences, and it looks like this year is going to be no exception.

Unfortunately, I will not be able to make either of those shows due to another commitment. But if you want to see my “oh” face in person, come to the “Oh Ess Em Cee” conference in November.

Last year all three conferences were held the same week, which was very disappointing for me as it was hard to choose which to attend. This year the Open Source Monitoring Conference was pushed out a month and will be held in Nürnberg, Germany, 18-20 November. While mainly thought of as an Icinga and Nagios conference, the organizers have been very inviting of other projects. We have had a presence there for the last couple of years, but I have only personally been once and it was amazing. So many people sign up that they are able to pretty much rent out an entire hotel, so while the conference is always good it is the conversations outside of the presentations that are the most enjoyable.

I’ll be giving a talk on OpenNMS (‘natch) as well as getting up to speed on what else is going on in the monitoring world.

I hope you can make at least one of these shows. You won’t regret it.

Why There Will Never Be Another Red Hat

October 13th, 2014

My friend Nick sent me a link to a post called “Why There Will Never Be Another Red Hat: The Economics Of Open Source“. It immediately pushed a bunch of my buttons before I read the first word of the article.

First, it was from TechCrunch. I have nothing against TechCrunch and I respect a lot of their work, but they are Silicon Valley-centric, if not the main mouthpiece, and thus I have to take that bias into account when reading their articles. What works in the Valley doesn’t necessarily translate to the world as a whole, which is why a lot of Valley companies seem to quickly plateau after an initial success.

Second, continuing the theme of Valley biases, I strongly believe Red Hat doesn’t get the respect it deserves because it is headquartered down the road from us in North Carolina and not California. There is a strong sense of “you can’t make it if you aren’t here” in the Valley and that extends to a somewhat dismissive view of Red Hat. Plus, I have a big ol’ man-crush on Jim Whitehurst as he is the most successful tech CEO I know that really, really “gets” open source and thus anyone trying to tell me about the “economics of open source” without respecting Red Hat starts off on my bad side.

Finally, the author is currently at the VC firm Andreessen Horowitz, or A16Z as those of us in the know refer to it. In last year’s investment tour I met with a large number of people in the Valley, and the guy I met from A16Z was easily the worst of the bunch. He made the caricatures of the Silicon Valley TV show look mild. He had no interest in us since we weren’t in California and he was more concerned with who we knew than what we did. I left that popularity crap behind in high school. Granted, we only rated an audience with the lower levels of the company, which overall does have a pretty solid reputation on Sand Hill Road, but still it was almost insulting.

Let’s just say my bullshit meter was halfway to pegged before I started the first sentence.

However, I found most of the article to be spot on. In my “Is Open Source Dead?” post I talked about how open source is both greatly increasing while the classic ideals of open source (i.e. free software for everyone) seem to be going away.

My own philosophy is that, at least for certain large and complex (i.e. expensive) software, the proprietary software model is doomed. Customer needs are changing so fast that no closed system can really keep up, and we’ve seen that in the biggest OpenNMS customers. I spent the last week in Ireland and the client was complaining that before they started using OpenNMS, whenever they needed some new functionality in their management solution the proprietary vendor took too long, cost too much and delivered too little to be worth it. Using a free platform like OpenNMS made it much easier to adapt the tool to their business workflow, instead of having to change it to meet the workflow of the management software. There is value in that – value that can be monetized.

Peter Levine started to win me over with “Red Hat is a fantastic company” (grin) and I as I read on I found my head nodding in agreement. He states

Unless a company employs a majority of the inventors of a particular open source project, there is a high likelihood that the project never gains traction or another company decides to create a fork of the technology.

While I estimate OpenNMS has around 40 to 50 active contributors, at least 15 of those are on my payroll, either directly or indirectly as contractors. While I definitely would like to increase the overall number, we are growing fast enough that we can usually hire someone who contributes a lot to the project, and then, since they can spend their full time on it, we as a group continue to contribute greater and greater amounts of the overall code. When I started out on my own back in 2002, I think at least half of the code came from outside of the .com side of things. Now it is probably closer to 5% or less. It has managed to let us focus on our direction for the application.

Then Levine continues:

To make matters worse, the more successful an open source project, the more large companies want to co-opt the code base. I experienced this first-hand as CEO at XenSource, where every major software and hardware company leveraged our code base with nearly zero revenue coming back to us. We had made the product so easy to use and so important, that we had out-engineered ourselves. Great for the open source community, not so great for us.

I’ve heard this tale from a number of people. Become successful and someone like IBM could dump 100 developers on your project and take it over. While we haven’t experienced it directly (rarely do people tell me OpenNMS is “easy to use”), we are constantly finding out about companies who have either based products on OpenNMS or used OpenNMS to provide services for profit. I think this is great, but it would be nice to capture some of that effort back into the project, either in the form of contributions or cash. It is one reason that the next major release of OpenNMS will be published under the Affero GPL.

So, with all this doom and gloom, what is the solution? Levine’s answer is “sell open source as a service”. I couldn’t agree more. This is exactly what we pitched to A16N. It’s something of a “win/win”.

This recipe – combining open source with a service or appliance model – is producing staggering results across the software landscape. Cloud and SaaS adoption is accelerating at an order of magnitude faster than on-premise deployments, and open source has been the enabler of this transformation.

If you have the in house development staff to leverage open source, it makes sense to become active in the community. David just finished a five stop roadshow in both the US and Europe describing the OpenNMS roadmap over the next year as we position the product for the Internet of Things. He met with our largest customers and all of them are eager to get involved, many pledging, OpenDaylight style, to provide the project with developers. They get input into the direction of the product and we get great open source code.

But what if you aren’t a large medical information company or a worldwide financial institution? You may need what OpenNMS can provide but don’t have the time to build in the workflow or customize it. We will have a solution for you.

The only issue I ended up having with the article was when he compared Red Hat to Microsoft, Oracle, Amazon, etc. Sure there might not be another Red Hat, but I don’t expect to see another Microsoft (operating systems and office suites), Oracle (enterprise software), or Amazon (online product distribution) either. This doesn’t mean that there won’t be new mega-companies. In ten years I expect there will be some huge companies that no one today knows about, probably in the areas of 3D printing and biotech (specifically integrating tech into the human body). Enterprise software companies will be represented by a number of large-ish but nimble companies leveraging open source, and I wouldn’t count out Red Hat just yet as they too are pivoting to follow this new business model.

I want to close with one little story. I spent last week in Ireland where I just happened to be in Dublin where the band Wheatus was touring with my friend Damian (MC Frontalot) opening. They had a grueling schedule (shows almost every night) but I managed to see their concert and talk with them afterward.

Damian has just released a new album (it’s excellent, buy it) and he closed his set with “Charity Case” which includes the lyric:

It’s true:
Frontalot’s destitute.
I need you
to buy my CD so I can buy food.

He often pokes fun on the changing nature of the music business (his song “Captains of Industry” suggests that he is not a musician but instead is in the T-shirt business), and we talked about various business models. I pointed out that acts like Elton John are living the high life from work that basically peaked in the 1970s, and that sort of “royalties for life” model is gone. In its place are artists who sell directly to their fans, often including personalized premiums for a higher price, and touring. The band Phish tours extensively and they make millions, all the while encouraging their fans to bootleg their music, something old school musicians wouldn’t think to do.

At this point Brendan, the founder and front man of Wheatus, joined us and stated “there is nothing an old musician can teach a young one about the music business”. That quote really resonated with me.

In summary, I really enjoyed this article. It mirrored a lot of our thoughts over the last year as we seek to make OpenNMS successful. Remember, our plan is for OpenNMS to be *the* de facto management solution of choice and for that to happen will take a lot of work as well as money. But one thing that we will continue to do to emulate Red Hat is to keep publishing as much software as possible under an OSI-approved open source license.

That is still a key to our success: OpenNMS will always be free and OpenNMS will never suck.

Using an XML Parser

October 9th, 2014

You know when the XML nerds say not to use regular expressions to parse XML? They’re right.

As part of a less is more project, we wanted to remove the tags from all of the OpenNMS event files. We spent much of the morning playing with a number of methods to find and replace with empty space those tags, and we failed. We came close a couple of times, but then some weird aspect of formatting (tags that spanned multiple lines, some with spaces and some without, etc.) would foil it.

Then I found out about xmlstarlet. We installed it and ran:

xml ed -L -d "/events/event/alarm-data" [filename]

and it just worked. Pipe that bad boy through find and you are good to go.

While I don’t think the option exists, it would be cool if instead of deleting the tag we could just comment it out, but that doesn’t seem to be currently possible.

When Less is More

October 7th, 2014

One of the things I’ve noticed in my years of deploying network management solutions is that people can get real excited when they go from having no visibility into their network to being able to see in great detail what’s happening, as when they deploy OpenNMS. The problem then changes from having no information to having too much.

Network geeks like myself tend to be loathe to turn off certain alerts, but sometimes that can be the best thing for an organization.

When OpenNMS was started, workflow was based on events. Events appeared in the browser, events triggered notices, you could acknowledge events – pretty much everything was events. But events can be noisy, especially if you leverage the SNMP trap capability of many devices. This is why we implemented the alarms subsystem. Alarms can take many events and reduce them into a single alarm. Alarm processing can be automated to insure that issues that are important are escalated, and issues that have been cleared can be removed. The alarms list is supposed to be a “to do” list for the NOC staff.

In order to make that happen, it is a good idea to consider each alarm in your system and insure that it is “actionable”. Each alarm comes with two fields for tracking the resolution progress, and these can be used to document the actions taken to fix the alarm.

The “Sticky” memo field is used to annotate a particular instance of an alarm. For example, suppose there was a “link down” alarm due to a circuit being cut by a backhoe. The NOC engineer would be able to note that the repair was in process and maybe even include a case number. Once the issue is resolved the sticky memo goes away.

The “Journal” memo field is permanently associated with the alarm. This is for notes that could be useful the next time the alarm happens, such as “Contact Jim – he knows how to fix this”, etc.

Alarms can be acknowledged, which will remove them from the list of current issues. It is pretty easy to create an automation that can unacknowledge an alarm if it hasn’t been cleared in a particular amount of time. Thus you can automate “reminders” that the issue is still outstanding.

This doesn’t discount the value of events. In OpenNMS, events have become more like log messages. When an alarm happens on a particular node, that node’s page will reflect the events associated with it, which may shed some light onto the problem. But having too many events appear as alarms can overwhelm the NOC staff to the point that they stop using the system.

Unfortunately, often the best way of dealing with network issues involves trial and error. By limiting alarms it is possible to miss something important. But once that happens, alarms can be created to insure it doesn’t happen again. But the opposite, dumping too much information into alarms, will guarantee that alarms will be ignored, greatly increasing the chance that something important will be missed.

I developed my alarm philosophy during my first network management deployment in the early 1990s. I was consulting for a cellular provider and installing HP OpenView Network Node Manager (version 2.2 I believe) and they had me working in the server room. Besides being a bit cold, in the corner was a large UPS that was constantly beeping.

Beep … Beep … Beep

I asked Avery, the guy I was working with, what was wrong and he replied “Oh, it always does that”. At that very moment I decided that if there is an alarm and you don’t do anything to resolve it, just turn it off.

Just remember that OpenNMS is a platform and thus you get to make a lot of the decisions on how best to get it to work with your organization. Consider that when deciding which events to turn into alarms, and then focus on using automations to insure that the most important issues are treated as such.