Category Archives: Open Source

I Always Start with a KISS

Most of the time in tech, you can stand on the shoulders of giants and reuse great works to your benefit, whether those things be algorithms or engineering methods. But, sometimes, as you venture towards the edge, you realize that you’re now somewhere completely different. You find yourself floating in some unknown ether, and you need to reorient yourself to this new dimension. And the more I play with the Ethereum platform, the more I find myself in that position.

For example, when I first started to create the Wonka rules engine, I had a few plans in mind, with one being to possibly use the Rete algorithm within the project. After all, it provides an efficient solution for a dynamic situation. Specifically, this dynamic situation could be one where a rule at the end might change the data, and then the whole set of data would need to be reevaluated by the rules due to that change. In this case, the Rete algorithm provides a performant method for this possibly recursive set of reevaluations. But that’s in the normal world, not the Ethereum one.

Within most standard computing, there’s minimal cost to using a system’s resources. At worst, you’re paying a cloud provider for cycles; in order to run a limited set of rules, you’re still talking pennies. But in a blockchain environment like Ethereum, where you have a massively distributed database of decentralized shared resources, every execution is an expensively coordinated effort. In order to change the state of the system, the cryptocurrency of the system, Ether, is used to cover the cost of a transaction, which is described as the ‘gas’. And even though the market value for Ether keeps sliding down, it still means that every rule that changes the system is another line item on your ‘gas’ bill. That’s difficult to predetermine since it’s always based on context. Yes, you can run your own private Ethereum blockchain within your own network (instead of the mainnet), and that means you’re not paying real Ether at all…but, in the end, people using blockchains generally still want accurate measures of impending execution.

Which brings us to the ultimate question for anyone using a rules engine within a blockchain like Ethereum: how much is this damn thing gonna cost me to run? Since the answer could be “a lot”, we have to abandon our beautiful, time-proven algorithms and go back to the drawing board. Which is why when you’re dealing with nascent platforms, you have to go back to KISS (i.e., Keep It Simple, Stupid). So, in this MVP version of a rules engine, I did just that, creating a straightforward implementation that just gets the job done. In this current state, its destiny will more than likely be as unwept, unhonored, and unsung. However, at the very least, such a prototype can be used by others to evaluate and perhaps validate the existence of a rules engine in this new space. And, maybe, just maybe, it can serve as the first iteration of something that will eventually become worthy of note.

On a side note, it turns out that getting a ticket to DevCon is a lot harder than I thought.

Quick Tangent: Dumb Bots

So, there’s been so much talk about AI lately, and in particular, there’s a great deal of interest in bots. No, not the Mirai kind (which hopefully isn’t plentiful in the future, despite its Japanese translation). No, I’m talking about the friendly, enterprise kind. You know, the chatbots on Facebook that are supposed to be helpful snippets of AI, capable of booking hotel rooms for you. Of course, I don’t really understand the usefulness of these bots, since there’s no way that a bot could help me find the ideal room faster than my own investigation. In fact, they seem kinda…well…dumb. But these bots are probably not aimed at a self-appointed pariah of social media like myself. Instead, it’s probably meant for those people who are younger (i.e., millenials) and who are more predictable (troves of available marketing data via Facebook, less variety of purchases, etc.). In that case, I suppose that it’s useful for some but not for myself…or is it?

Similar to my reaction to chatbots, I never quite understood the newfound love for Slack. It’s a messenger app…so what? However, as I started to delve more into it, I started to understand its appeal through its extensible functionality, especially to developers. I can create a simple bot (or a basic web service) on my public-facing servers, so I can use Slack to talk with it on my phone and get the status of machines and processes? Okay…that’s kinda cool. (Assuming that your company and networking department embraces the idea of allocating machines just for this purpose. Trust me, I know…that can be a hard sell.) So, maybe, must maybe, I could be down with these chatbots. That way I could use Slack (or Skype) and be hip like the cool kids!

Hmmm…so how I could I actually pitch this one to the brass? Curious, I looked to see if there was already an enterprise version of such a solution, and though I did find one or two, they seemed to be costly and less flexible than desired. So why not just build one cheaply on my own? Since I recently read something about Microsoft’s nascent bot framework and its integration with Skype, I figured that I could start there as a quick way to prototype. After proceeding through a few quick tutorials, it became obvious that a chatbot is nothing more than a tailored RESTful web service, and with that realization, I quickly assembled and got working the prototype that I had in mind.

However, over the next few weeks, I started to realize that it wasn’t viable. One, since this framework is too young to even stand on its own wobbly legs, Microsoft keeps updating the framework and breaking my stable prototypes. As with previous experiences when dealing with a Microsoft gestation, I wondered again if Redmond’s new projects (along with their frenetic and seemingly bipolar updates) are victims of Conway’s Law…Two, I read about how Skype does not and will not support third-party bots that are not publicly registered in their Bot Directory. I’m fairly sure nearly all of the company brass would have a problem with a publicly available chatbot that tells the status of our internal servers. Just a hunch.

After taking a quick look at other platforms, I came away with similar impressions. In the end, I’d say that chatbots are like a lot of new tech these days: lots of potential but some distance away from ultimately being practical.

Quick Tangent : XML Schema Evolution

So, after my last post, I got curious: is there any software out there that performs XML schema evolution, even if it’s proprietary? Oddly, after searching for a few minutes, the answer “no” seemed to be coming back from the web. Now, Oracle and IBM do offer a service to update your current XML documents according to a new schema…but only if it doesn’t invalidate the old schema. Basically, their “evolution” functionality allows you to further refine your schema’s rules, like changing the maximum/minimum of a tag’s occurrence or adding a new required tag. That’s hardly any sort of evolution; it doesn’t even provide the ability to automatically rename tags/properties like Avro! So, the claims of Oracle and IBM might be more marketing than engineering.

But I guess that marketing and buzzwords are all too normal in software…After all, whoever coined the term string interpolation definitely took some severe liberties, since it’s sure a long way off from real interpolation. In any case, there seems to be an opening for a niche market here, one which could be somewhat lucrative. However, these days, all the big bets of towering chips are on the table of machine learning, big data, and AI. In the eyes of the major league, anything that deals with XML (i.e., old-school data processing) should go play the slot machines.

Good for me…I don’t mind being stuck alone in a dark corner! Reminds me of playing Street Fighter 2 by myself in the back of a pizza parlor and having a blast…In any case, I was looking for tools that could help build an engine for XML schema evolution. Interestingly, I found an open source project by Dmitry Pekar that can convert both ways between XML and Avro. That could help by extending the functionality already in Avro…but besides the simple renaming of tags/properties, it doesn’t satisfy my proposed requirements. (Plus, your distributed architecture would have to ultimately use Avro, which would be a refactoring headache in some instances.) I haven’t found anything else yet, which makes me suspect that my handcrafted MDD approach might be the only viable option.