While reading Slashdot news the other day (I know, I know…all the cool kids read Hacker News), I stumbled upon some news about RethinkDB that didn’t seem all that relevant to me at first. However, upon reading some of RethinkDB’s documentation, I realized that such a database might be another useful tool in a MDD-constructed engine for XML schema evolution. From time to time, I still think about an engine capable of enabling complex translations between different versions of a XML schema; essentially, this engine would generate and then use XSLT files based on the metadata inside certain database tables. If we put this metadata inside tables of a RethinkDB database, RethinkDB’s functionality of posting messages to a queue (when a table’s structure or data has been changed) would be a neat way to trigger the rebuild of these XSLT files!
Well, as I said before, I’d get back to metadata-driven design…and here we are!
So, as I was perusing InfoQ one day a few weeks ago, I stumbled upon an interesting video by Vinicius Carvalho of Pivotal. Basically, within the video, Vinicius (okay, I’ll admit it – it’s a cool name that I wish I had) addresses an issue familiar to anyone who creates web services : how does one evolve a payload’s schema without breaking the clients of users who referenced the old schema? For example, if we were returning a payload with a property/tag called ‘Price’ and if we wanted a new version of the schema to replace that tag with ‘PubPrice’, how could we do that without requiring every user to change their client/consumer app? These kinds of presentations are my favorites, since they address real-world problems.
So, in his presentation, Vinicius goes about demonstrating how one can create a solution to such a dilemma. Since he does work for Pivotal, he uses the Spring platform to present a scenario where a web service has an original schema that needs to be altered in its eventual evolution. (Granted, he’s probably a fan of the Spring framework, which means that he’s a fan of event-driven frameworks…but we won’t hold that against him. I’m kidding, I’m kidding…take it easy, Spring zealots.) For the first few minutes, Vinicius focuses on format, which is important when discussing web servers that return payloads. (And, yes, I agree with him: JSON is bad, mmmkay.) Even though I’ve never used it since we require verbosity (i.e., XML/JSON) from my stakeholders, he does make a compelling case for the Avro protocol; it does appear to be an impressive format, one that can be very powerful in capable hands.
However, the most interesting part is when Vinicius begins to talk about the actual mechanism for resolving the focus of this presentation: schema servers and their registries of schema versions. Basically, using features within Avro, a schema server allows the creator of a web service to register their original schema and any subsequent versions of it. When a new version of the payload’s schema is conceived, the new schema can be submitted to the schema server in order to test whether it breaks (i.e., is not backward-compatible with) any older versions of the schema. Plus, accompanied with markup language, the new schema can indicate any tags which will replace tags existing in the previous version. So, when an older client submits data to the updated web service, the web service can use the schema server as a translation device. Nifty!
I love this solution, and I want to commend Vinicius and his colleagues for sharing a solution to a common problem. However, what if we wanted to evolve this solution for more complex scenarios, where the changes to the payload are more involved? For example, what if we wanted to split one tag into several others? Or what if we wanted to replace an entire composite with another one? In this case, simple markup language wouldn’t be sufficient enough to indicate how the schema server could transform the data from one form to another. In this scenario, you would need to create a tool that could help you define such transformations systematically, and you would need the right methodology in order to build it. You might know where I’m going with this one…Yes, I think that this is where the application of MDD could produce the guts of the schema server and make it even more powerful!
Given, if we stayed with Avro, it would be difficult to create such a translation service; the functionality built into Avro is likely difficult to extend (if even possible). However, if we use XML (which, in my line of work, we are most apt to use) as our API payload’s vehicle, we could use something that Vinicius mentions in his talk: XSLT. Even though I can’t say that I never cursed when using it, it can be a helpful tool in certain cases…and in the case of creating a MDD server for schema translation, it fits perfectly! Using MDD, we could create a schema server that could generate the appropriate XSLT and then perform complex conversions from one XML schema to another. I have a few ideas on how to make such a thing work…but that’s for another time.
Over a year ago, I published an article that proposed the possibility of creating a system that could be configured to systematically download snapshots of records via a data API, catalog them for auditing purposes, and then apply that data to relational tables used in production. At the time, there was only the article about the subject matter, but there wasn’t a working example to accompany it. In order to further demonstrate the feasibility of the article’s proposal, I decided to go ahead and remedy that particular dearth.
So, after taking a few minutes out of each day to incrementally build the project, I think that the project has come along well enough to be let loose upon the world. I’ve uploaded the project to GitHub, so that it may endure for posterity. For the purpose of inspiration or amusement through ridicule, only time will tell. 🙂
In the last post, we had finally finished gathering all of the metadata ingredients required for our recipe, and we were now planning on our cake’s final layer: the permissions algorithm. Of course, we could simply code the algorithm in the flowchart displayed beforehand in Part 1, using the metadata in a few modules to enforce our security paradigm. And then we could call it a day. Sure, we could finish there…but that would be an antithetical conclusion to all of the work done up to this point. After all, a major argument for MDD is to reduce the need for recompilation and/or redeployment of software; we’ve already used it to create an unique ORM framework that has been leveraged by such a design. So, instead of cementing these rules of our permissions algorithm into our code base, could we make them more dynamic as well?
As it turns out, yes, we can. In a previous InfoQ article, I described how one could create a business rules engine driven by a simple pidgin, a black box with a lexicon whose terms are determined by the very same metadata that drives the main infrastructure of the architecture. In addition to business rules, we can take advantage of the same approach when we want to implement the permissions algorithm, and for the sake of clarity, we will call this black box a permissions engine. We can then compose an enterprise DSL that incorporates both the Attributes and the contextual data describing our records. Since it should be comprehensible to a literate business user, we should design it with the priority of being easily reviewed and maintained by various stakeholders. As an example, let’s handle the scenario depicted in the flowchart from Part 1 by creating and showcasing a snippet using such a DSL:
In this sample (and much like the scenario described in the InfoQ article), our engine expects two records to be supplied in order to successfully execute the permissions algorithm. There is the Current instance of a record and its Attributes on the system, and there is an Incoming instance of the record with new data (which has been submitted by IncomingUser). While the overall system has the responsibility of loading and then passing along these relevant records, it becomes the duty of the permissions engine to understand and then enforce the rules of this presented sample. In this case, our permissions engine will use both records and any potential contextual data to determine whether or not to save the incoming “Price” Attribute.
Even though it is possible that the “Price” Attribute deserves special consideration, it’s more than likely that we will want to treat most or all of these Attributes in the same manner when processing a single record. In that case, we could use this sample as our default algorithm and wish to associate it with many (if not all) of our Attributes. By substituting “Price” with a placemarker that will eventually be replaced by our target Attribute name, we could reuse this same DSL snippet above and eliminate the pointless redundancy of multiple copies. However, if there are indeed special cases, we still have the option to create separate algorithms. All of these algorithms can then be stored as documents within an optimal retrieval database (like MongoDB) and pulled later by the permissions engine during its initialization. With just a bit of organization, we are now able to create a truly customized permission system for each data point that is submitted to the system.
With our permissions engine, this subsystem will then determine the applicability of each data point and whether it has the clearance to be persisted into our database. On top of this important functionality, we can reuse this subsystem in a number of ways. It’s not outside the realm of possibility that certain qualified stakeholders might want a preliminary report on the levels of clearance for their data before submission, so that they can get a preview of its fate. (For those who are fans of the movie “Minority Report”, the term precog certainly comes to mind.) In some cases, such intel could prevent wasted attempts at saving certain payloads (which can be a time-consuming process of multiple validation and correction phases) and could give those stakeholders the chance to address whatever potential obstacles might block their way. For example, Bob Smith might have the price locked on a few records, and Sue Doe might need Bob to lift the lock temporarily in order to submit a few emergency updates. In that case, we could create a user-facing service that simply invokes this subsystem, in order to generate and return such a report to the calling stakeholder. In that way, Sue does not need to learn about this discrepancy minutes later, when she receives the final status of her submitted record batch; she can quickly get this information beforehand, saving valuable time.
At the start of this series, it was my intent to present the feasibility of creating a permissions subsystem using MDD. Hopefully, at this point, I have at least shown that. With some ingenuity, we could likely repurpose this subsystem for yet other uses. Better still, we can clone this subsystem and adopt it as a solution for other similar dilemmas. It would fulfill my youthful dream of applying such a method to a filesystem, and I still think that such a strategy is possible. However, I will leave the onus of such a chore to a younger version of myself with more spare time…or, better yet, to posterity.
You can find the first post in this series here.
Since we are discussing the specifications that could drive such a subsystem of an architecture, we need to first identify a methodology that provides optimal benefits when building it. We need a design method that provides both a great deal of flexibility and incorporates a granular approach to dealing with data. In such a case, I would turn to metadata-driven design, on which I have expounded in the past.
So, what is metadata-driven design (i.e., MDD)? For the sake of brevity, MDD can be thought of as an increment to domain-driven design, where metadata provides the blueprint for the storage, the data structures, and the functionality inherent to an enterprise-scale application. Through the addition of more rows of metadata, stakeholders can extend the scope and functionality of the application with little to no additional software development. However, if any actual software development is required to enhance the platform due to some unforeseen complexity, it should not present that much difficulty; this increment should also be able to use additional sets of metadata, existing as extra layers on top of the original set(s). These additional layers can be thought of as dimensions, and much like a communications protocol, the set of these layers can be thought of as a stack.
So, let’s showcase an artifact from the InfoQ article that started it all, which provides an introduction to MDD:
In this image, we have an example displaying a set of metadata that describes a logical group of data points (i.e., Attributes); we can address them as a group, especially since they reside on the same table. Using this metadata, we could generate static data structures, but by using “flexible” structures, we can gain the benefit of not having to recompile or redeploy any of the code for our servers. So, we will create flexible data structures that act more like a series of nested containers, using the metadata to determine the hierarchy of this container.
For example, these attributes could be collected in a hash table of “[GroupName] > [Attributes]”, where the key “GroupName” is a string and the value “Attributes” is another hash table; the “Attributes” hash table could contain the actual pairing of each Attribute to its value. This particular metadata helps to construct the base layer of our stack. On top of that initial dimension (and presented in the bottom of the image), we have a new set of metadata that describes permissions of users in relation to each Attribute; by doing so, we have created the preliminary parameters for our permissions schema. However, as stated earlier, we need more information in order to have a more complete perspective.
So far, we have a way of packaging the actual data, and we have a way of storing user permissions in relation to the data points. Now, in order to finally create our permissions subsystem, we need a set of information that describes the state and history of each data point. Let’s assume that whenever we persist data to our system, we also write records to an auditing repository that describe the events that have just occurred. (After all, it’s recommended to have such a recorded history on hand; it’s our protective shield in the face of the dragon known as SOX.) For example, if the stakeholder Bob Smith changed a price from $3.99 to $2.99, we would log just that, along with any other edits from years prior. When we need to know who made the last edit for this product’s price, we could scour this huge table, with many rows listing such details from months and years ago…but for the advantage of performance and general reliability, we should designate a place to put particularly vital information, like data about the most recent edit. So, we will create yet another dimension to the metadata and add it to our stack.
Much like the initial dimension that described the structures of the actual data, this dimension will describe a structure that represents the state and history of each instance of an Attribute (i.e., per record), and this information will also be persisted to a table. We will call this information contextual data:
Now that we have a definition of contextual metadata, we can start to employ it when we persist main records, writing context records in parallel. Take the following set of contextual data as an example instance, which follows the definition in our metadata:
This contextual data lists a number of important properties regarding the “Price” Attribute on the record with EAN 1234567890. It tells us that Bob Smith locked this record’s price on 5/10/2016, and it tells us that Bob Smith also edited the price on the same day. Using some simple queries, one can find this data within our immense auditing repository. However, by updating this simple table while simultaneously depositing to the huge vault of our auditing, we can improve performance by having such pertinent information quickly returned via simple queries. Now that we have our final, requisite ingredient for our recipe, we can finally take the steps to create our desired subsystem, of which we will do in Part 3.
When my career was nascent many moons ago, I worked at a financial company as an intern, and since there was a good deal of work and too few occupied seats, every employee was expected to wear a few hats at any given time. In those first days of my service, the majority of my day was filled with the tedious work of testing our services as QA. Occasionally, I was allowed to participate in development, but most of my extracurricular activities (i.e., rewards for good behavior) constituted being an administrator for our development and QA machines.
Unsurprising to any seasoned developer, the knowledge attained from being a system administrator enhanced my general skillset by a magnitude or two, and those lessons were crucial to all code that I subsequently wrote in the years to come. In addition to being exposed to the awesome power of scripting and the idiosyncratic differences between different filesystems, one also learns the subtle complexity of something that seems initially simple: permissions. Whether it pertains to setting file permissions via ‘chmod 777’ or to allocating roles in a Windows domain, one can observe that authorization resembles an art form, where you learn to balance the needs of your users with the requirements of security policies.
However, even as a young apprentice, it wasn’t long before I wondered about the possibility of an even better solution to the various problematic scenarios encountered as an admin. After all, there are a number of situations where a simple permissions mask just doesn’t suffice. Could it be possible to build a more detailed solution, like setting permissions on sections of a file instead of the whole file? Before allowing an action, could this subsystem weigh levels of trust belonging to different users and their relativity to one another? What if such a subsystem could assign security policies to a user that took context into account, like a policy that allowed users to create a file but never update or delete them? Decades ago, it was just a passing thought of a cadet in training, but it would come back to the forefront of my conscience when we had to design an architecture that required something similar to just that.
When it became necessary to build the next iteration of our production system, we dove into the new specifications, and based on the various requirements and requests, we became aware of the dire need to reimagine just what was meant by the term permissions. In our situation, we needed to create a distributed architecture and the inherent software that would allow users familiar permission settings, for activities in our database (i.e., CRUD)…but that was only the surface level. Found pervasively throughout large-scale corporate environments, a fair amount of scrutiny is necessary in order to ensure that each submitted data point belongs within the system. In some cases, you can build customized solutions for a specific platform, much like Grzegorz Gogolowicz’s proposal regarding Sharepoint. Other times, though, you need a more abstract, granular approach where each section of a data record should have some level of clearance, just as when protection keys are used to secure memory pages in an operating system. In our situation especially, some sources (data vendors, stakeholders, etc.) were more considered more reliable than others when it came to accepting certain pieces of data, and the current context of the data only complicated matters further.
For example, Data Vendor Inc. could be relied upon to provide an initial price, but if stakeholder Bob Smith sends a subsequent price, we should overwrite the vendor’s submission with their data since we trust Bob Smith more. In this scenario, the architecture would then need to examine the context of the situation, taking into account various factors: the previous editor of a column, the current user, the current state of the data, the metadata about that column, etc. In the end, this permissions mechanism became a more intricate filter, one that required a number of provisions in order to make a proper evaluation.
We could have created a sophisticated set of libraries with accompanying configuration files, but it seemed too rigid and troublesome, especially when we thought about the disaster awaiting an eventual expansion of the system (additional columns, additional stakeholders, etc.). In short, we needed an approach to our resolution that could easily and deftly dodge any of these hurdles thrown at it, so that we could save ourselves the headaches of a potential security and auditing nightmare. And after some deliberation, we found our direction, and hopefully for readers, it will find an application outside of just our environment. We’ll find out with the publishing of Part 2.
In my fourth professional article, I describe how metadata-driven design can be employed to create a dynamic DSL that serves as a business rules language, one simple enough to be a tool for your company’s users. Again, I must give many thanks to the staff at InfoQ for taking the time to help with this submission. If you support business users who are constantly involved with the daily ingestion and maintenance of enterprise data, they might just thank you if you create a DSL for them!
Metadata-Driven Design: Creating an User-Friendly Enterprise DSL
Aaron Kendall shows how to build a domain-specific language for a saavy but non-technical crowd which enforces the rules that apply to our business domain.