While reading Slashdot news the other day (I know, I know…all the cool kids read Hacker News), I stumbled upon some news about RethinkDB that didn’t seem all that relevant to me at first. However, upon reading some of RethinkDB’s documentation, I realized that such a database might be another useful tool in a MDD-constructed engine for XML schema evolution. From time to time, I still think about an engine capable of enabling complex translations between different versions of a XML schema; essentially, this engine would generate and then use XSLT files based on the metadata inside certain database tables. If we put this metadata inside tables of a RethinkDB database, RethinkDB’s functionality of posting messages to a queue (when a table’s structure or data has been changed) would be a neat way to trigger the rebuild of these XSLT files!
So, after my last post, I got curious: is there any software out there that performs XML schema evolution, even if it’s proprietary? Oddly, after searching for a few minutes, the answer “no” seemed to be coming back from the web. Now, Oracle and IBM do offer a service to update your current XML documents according to a new schema…but only if it doesn’t invalidate the old schema. Basically, their “evolution” functionality allows you to further refine your schema’s rules, like changing the maximum/minimum of a tag’s occurrence or adding a new required tag. That’s hardly any sort of evolution; it doesn’t even provide the ability to automatically rename tags/properties like Avro! So, the claims of Oracle and IBM might be more marketing than engineering.
But I guess that marketing and buzzwords are all too normal in software…After all, whoever coined the term string interpolation definitely took some severe liberties, since it’s sure a long way off from real interpolation. In any case, there seems to be an opening for a niche market here, one which could be somewhat lucrative. However, these days, all the big bets of towering chips are on the table of machine learning, big data, and AI. In the eyes of the major league, anything that deals with XML (i.e., old-school data processing) should go play the slot machines.
Good for me…I don’t mind being stuck alone in a dark corner! Reminds me of playing Street Fighter 2 by myself in the back of a pizza parlor and having a blast…In any case, I was looking for tools that could help build an engine for XML schema evolution. Interestingly, I found an open source project by Dmitry Pekar that can convert both ways between XML and Avro. That could help by extending the functionality already in Avro…but besides the simple renaming of tags/properties, it doesn’t satisfy my proposed requirements. (Plus, your distributed architecture would have to ultimately use Avro, which would be a refactoring headache in some instances.) I haven’t found anything else yet, which makes me suspect that my handcrafted MDD approach might be the only viable option.
Well, as I said before, I’d get back to metadata-driven design…and here we are!
So, as I was perusing InfoQ one day a few weeks ago, I stumbled upon an interesting video by Vinicius Carvalho of Pivotal. Basically, within the video, Vinicius (okay, I’ll admit it – it’s a cool name that I wish I had) addresses an issue familiar to anyone who creates web services : how does one evolve a payload’s schema without breaking the clients of users who referenced the old schema? For example, if we were returning a payload with a property/tag called ‘Price’ and if we wanted a new version of the schema to replace that tag with ‘PubPrice’, how could we do that without requiring every user to change their client/consumer app? These kinds of presentations are my favorites, since they address real-world problems.
So, in his presentation, Vinicius goes about demonstrating how one can create a solution to such a dilemma. Since he does work for Pivotal, he uses the Spring platform to present a scenario where a web service has an original schema that needs to be altered in its eventual evolution. (Granted, he’s probably a fan of the Spring framework, which means that he’s a fan of event-driven frameworks…but we won’t hold that against him. I’m kidding, I’m kidding…take it easy, Spring zealots.) For the first few minutes, Vinicius focuses on format, which is important when discussing web servers that return payloads. (And, yes, I agree with him: JSON is bad, mmmkay.) Even though I’ve never used it since we require verbosity (i.e., XML/JSON) from my stakeholders, he does make a compelling case for the Avro protocol; it does appear to be an impressive format, one that can be very powerful in capable hands.
However, the most interesting part is when Vinicius begins to talk about the actual mechanism for resolving the focus of this presentation: schema servers and their registries of schema versions. Basically, using features within Avro, a schema server allows the creator of a web service to register their original schema and any subsequent versions of it. When a new version of the payload’s schema is conceived, the new schema can be submitted to the schema server in order to test whether it breaks (i.e., is not backward-compatible with) any older versions of the schema. Plus, accompanied with markup language, the new schema can indicate any tags which will replace tags existing in the previous version. So, when an older client submits data to the updated web service, the web service can use the schema server as a translation device. Nifty!
I love this solution, and I want to commend Vinicius and his colleagues for sharing a solution to a common problem. However, what if we wanted to evolve this solution for more complex scenarios, where the changes to the payload are more involved? For example, what if we wanted to split one tag into several others? Or what if we wanted to replace an entire composite with another one? In this case, simple markup language wouldn’t be sufficient enough to indicate how the schema server could transform the data from one form to another. In this scenario, you would need to create a tool that could help you define such transformations systematically, and you would need the right methodology in order to build it. You might know where I’m going with this one…Yes, I think that this is where the application of MDD could produce the guts of the schema server and make it even more powerful!
Given, if we stayed with Avro, it would be difficult to create such a translation service; the functionality built into Avro is likely difficult to extend (if even possible). However, if we use XML (which, in my line of work, we are most apt to use) as our API payload’s vehicle, we could use something that Vinicius mentions in his talk: XSLT. Even though I can’t say that I never cursed when using it, it can be a helpful tool in certain cases…and in the case of creating a MDD server for schema translation, it fits perfectly! Using MDD, we could create a schema server that could generate the appropriate XSLT and then perform complex conversions from one XML schema to another. I have a few ideas on how to make such a thing work…but that’s for another time.
“Since when did this blog become solely about indoor navigation? I thought that this thing was supposed to be about metadata?” Well…I can’t argue with you. I need to get back into that at some point.
In any case, after sampling different platforms for indoor navigation, I’ve come to notice something: there is no open source standard for indoor navigation yet. Of course, there is much discussion about indoor navigation within the gaming industry, especially within the community for Google’s Project Tango…but there isn’t as much talk when it comes to API standards. Considering that open source standards seem to be falling from the sky in the last few years (for cloud computing, for automobiles, etc.), it seems fitting that one of the bigger players (like Indoor Atlas or Estimote) should take this opportunity to lead the way with an open standard. As IoT invades our lives, indoor navigation will probably become more prevalent, and more development standards would be beneficial to the industry. (Of course, these Northern Europeans are probably too busy slugging it out, since that’s part of the travails of being a startup company.) I suppose another unmentioned company could also take this lead, but I’ve found that many are not as developer-friendly as Indoor Atlas and Estimote; most require any interested parties to fill out an application before even allowing access to their documentation.
Now, it wouldn’t have to be an all-encompassing standard, but it should probably take into account each of the strengths in the current set of available platforms. In addition to the practice of emulating Apple’s Location Manager (which they all seem to do in their iOS SDKs), an open standard could include interface methods for functionality like:
- The ability to overlay the indoor navigation map over an actual world map (which is offered by Indoor Atlas).
- The ability to generate a map of the indoor space dynamically (which is offered by Estimote).
- The ability to raise a signal (email, text, etc.) when someone enters the navigable area (which is offered by both Indoor Atlas and Estimote).
- The ability to generate an account with the vendor’s services programmatically (which I could see as being helpful to developers who want to incorporate these services into their own products).
Maybe I’ll create a sample and upload it to Github in the near future, as a more verbose example.
On a side note to this new standard, we can leave out some of those confusing and misspelled messages that are generated by Apple’s Location Manager. Who comes up with these things?
Oh, I had such lofty aspirations when I came up with the idea for a haunted house game. I had been so sure that there was an existing technology whose SDK would fit my plan…and, at this point, maybe there is!
So, having found an indoor navigation system that didn’t require the purchase of peripheral beacons, I decided to actually give it a try (though, due to past experience, I didn’t get my hopes up). So, I followed their tutorial’s advice, and I constructed a map image of my home. I created the map to scale, in order to achieve optimum results. After I uploaded the plan to my Indoor Atlas account, I then proceeded to follow the next step: using their tracker app, I did two walkthroughs of my house, hitting over a dozen waypoints with various permutations of edges. (After all, you’re creating a directed graph for them.) With some assistance from the helpful people at Indoor Atlas, I got my settings and floor plan to be exactly as I needed them to be. Thanks Elina!
Since they didn’t have an app that helped demonstrate the power of their SDK, I sought an open source project that helped to showcase their tech, and after poking around a bit, I eventually found one. I pulled the project, and after making a few corrections and downloading the right parts (updates from Apple, Homebrew packages, required Cocoapods, etc.), I was able to compile and deploy the project to my iPad. (I have to admit that I love how XCode offers to fix your code when certain methods have been deprecated or renamed. I wish that other IDEs offered the same service.)
When I ran the demo, it did load my map image, and I was able to navigate around my place with the demo project in hand. Finally…I had achieved some form of success! Now that I had a feasible technology stack as a solution, I could reflect on this platform as a possible choice:
- The indoor-navigation functionality works well, even though it’s sometimes inaccurate in more confined spaces and smaller rooms.
- The SDK is fairly easy to understand and integrate into your project.
- In order to integrate their tech stack into your project, there is an investment required, especially for any potential user of your product (generating a scaled map image, walking through your place, etc).
- Your potential product will require the user to create an account with Indoor Atlas.
In the end, I’m not exactly sure if this particular solution is a best fit…but any potential is good news to me. I’ll have to play with it a bit more in order to find out.
Well…I might be a little “tricksy” in that announcement, since it might not be exactly what you think. No, I haven’t yet ported the solution to Swift. (After playing with a few projects pulled from Github, I noticed how different Objective-C and Swift are from the state of iOS development 5 years ago. Seems like there might be a little bit of a learning curve there.)
However, the good news from Microsoft with .NET Core keeps coming. So, on top of delivering the port to Linux, they released the preview of Visual Studio for Mac only a few weeks ago. And the reaction seems to be generally positive! Now, all Mac-centric companies that deal with book data can make use of my ONIX Data Library. We just welcomed another 7 people to the fold!
So, as for the waiting addressed in the last post (i.e., the latest about the Haunted House saga), it ended when the next package of beacons finally reached my house. Almost immediately upon arrival, I registered the beacons using my Estimote account, and now having a total accumulation of six beacons, I placed them around my home and started the Indoor Location app on my iPad. As excited as Ralpie Parker with his BB gun in A Christmas Story, I started the app and followed the tutorial. “Okay, now I just need to create a location with the ‘Add new location’ button…where is that button…” After a minute of being unable to find it, I looked online and found nothing to assist me, feeling a little like a newb for not being able to resolve my problem.
Defeated (and now feeling more like Ralphie when he shot his eye out), I sent an email to Estimote so that they could enlighten me. Now, they did respond quickly with an explanation…but it wasn’t exactly the one that I had wanted to hear. It seems that the iPad could only navigate an indoor location with their app; if you wanted to setup a location with their app, you needed to have an iPhone. “Say what?” Yes, you’ll want to navigate a location using an iPad’s wide display, but you’ll have to create the location with the phone. It is, for the most part, a two-device solution. (My only guess is that the Bluetooth capability of the phone is more powerful than the pad.) Of course, one can imagine how happy that answer made me, but in fairness to Estimote, they empathized with my plight by refunding the cost of the beacons to me. Still, I would recommend that they explain the hardware requirements in flashing red letters on their site for future customers, since that might be something of interest to them before making a purchase.
So, back on the hunt I went. I was definitely finished with looking for a hardware-specific solution, and after looking around a bit more, I found a compelling product that needed no peripheral hardware at all: Indoor Atlas (i.e., IA). Interestingly, their solution did not need beacons since instead of using Bluetooth or GPS, it relies mainly on the electromagnetic fields that exist within the confines of a building. Based on my limited time spent reading (i.e., scanning every other word) and as I understand it, it seems that each room has an unique signature in terms of its EM field, and if you walk around your appartment with scattered waypoints, it can create a traversal graph based on your walk, with the unique EM fields being nodes. (If some WiFi is available on the premises, that can be of some help as well.) And all you need is your phone and nothing else. Cool! Even though I’d been let down so many times before, I got excited once again, and I downloaded their Map Creator app in order to get crackin’.
So, creating a map went fairly smoothly, with me walking from waypoint to waypoint in order to exhaust all permutations of possible navigation. In fact, as you create all possible edges between vertices in this messy graph, the app shows you its estimation of where you are, and with each new edge, it does seem to get smarter. Once I was done with my walkthroughs, I uploaded the data from the MapCreator app to my Indoor Atlas account, and I proceeded to generate a map of my place. The only question was: how can I use it? With Estimote, they had an app to showcase their tech, so that you could watch it in action before writing one line to call their API…but not so with IA, it appears. Basically, if you want to navigate your generated map, you’re going to need to get dirty: create your own app and access the map through the Indoor Atlas API. Since IA supports both Android and iOS, I was hoping for some sort of Xamarin solution…but, unlike Estimote again, no dice. It looks like I’ll be blowing the dust off my buddy’s aging MacBook in order to see what I can cook up on my own.