Where Have I Been?

Posted by David Estes on Jul 03, 2017

Filed under Grails, Open Source, Morpheus

Hello Everyone,

It's been a long time since I have had the time to stop and write a blog post and is something I regret taking the time to do. I'm still alive and well, actively contributing to the Grails / Groovy community (though not as actively as I was before). The reason for this is because of my engagement in a start up that has kept me busier than I could have ever imagined it would at the time we started. I have been working on a project called Morpheus which is a Cloud Management Platform (CMP for short).

Morpheus originally started as an internal project for our companies while working at Bertram Labs as a means for us to more easily manage and optimize these company's IT teams. It was even originally founded as a SaaS based offering for DBaaS (Database as a Service). But it is a much different animal than it was 4 years ago. When we set out to build a tool that made it easier for our teams to develop and deploy applications across a wide array of architectures (both in the private and public cloud sectors) we knew it was going to be ambitious. We had several goals in mind when we wanted to build it. Not only did we want to be able to provision databases at the click of a button , we wanted those databases to be properly configured for high scalability at the outset. This was not going to be a playground or "dev" environment. This was going to be a fully maintainable production deployment environment. And, it was going to be built leveraging our experience in Groovy and Grails.

As many of you know, our team has contributed several plugins to the open source Grails community (asset-pipeline, karman, selfie, spud, force-ssl, distributed-lock, security-bridge, etc.). We wanted to be able to take advantage of all that hard work as well as keep supporting these plugins on large scale applications. This project leverages almost every major feature of the Grails 3 framework including server side rendering with GSPs (with performance that feels almost like its client side) as well as JSON Views for our public developer API, and, of course, asset-pipeline. Based on this stack some may wonder why we didn't do a SPA approach to begin with and while I believe we will eventually move that way, we had very good reasons at the time for not moving into a SPA approach which was mostly time. We would have rather built out a prototype product than spent time fighting with a client side framework and all the complexities that come along with it.

What is a Cloud Management Platform?

The concept of a CMP is new to some people and can bring about some confusion so let me clarify what a CMP is. CMP is a tool that integrates with many cloud services (both private and public), and provides a clean consistent means for managing assets running within those clouds and provisioning new ones. It basically ties all the pieces together that you might need. In most companies it is not enough to simply provision an instance in aws or on vmware. There are policies that govern how these companies operate including things like naming conventions, DNS registration, IP address management, installation and configuration of logging tools like Splunk, Monitoring services, and even Configuration Management tools like Chef, Puppet, Salt, or Ansible. Wow, that's definitely a lot of steps right? A CMP provides integrations and automation tools to make this all happen automatically when instances are provisioned and to provide self service to teams of developers while IT can still maintain visibility and policy enforcement. For a manager this sounds great right? What about for a developer? Why do CMPs matter for developers? Most developers just want to deploy some code to a server right? In some companies requesting servers to deploy their code can take weeks due to all the steps involved and people involved in the process. A CMP can make companies provide a self service portal that gives their engineers servers in minutes not weeks.

Now every company is different. Some are very laxxed on policies like this and utilize DevOps teams to help mitigate and manage application deployments across servers unhindered. Some are small companies that have a small group of engineers that wear both the hats of IT and developers. This product was originally and still is designed by a group that has more autonomy in deploying applications than other companies with strong governance and we use Morpheus literally every day (eating our own dog food). So why would a developer or small shop be interested in a product like Morpheus? This is where Morpheus stands out. We built a hollistic solution for cloud management. We didnt just build a tool that tied into expensive third party services like Splunk for logging, F5 for load balancing, Veeam for backups. While we do provide that, we also built our own implementations of those services. We also built code deployment management and ways to customize code pushes to servers, auto scaling capabilities, and even built in monitoring with alerting. Even better, we built a CMP that can be installed in 15 minutes on a blank linux VM. This is great for small shops that don't have a budget to invest heavily into all these tools and frankly dont even necessarily need some of the features these tools offer. I personally use Morpheus for all my self hosted projects. It sends me text alerts if a hosted server goes down and if possible even moves services to another server thats operational. It provides a very clean interface (one of the most beautiful looking web apps I've been involved with and used but I may be biased). And a CLI so full of features an capabilities I rarely ever use the UI (259 commands available in the CLI at the time of this post).

How did it Start?

When Morpheus started, it started as an internal project within Bertram Labs. It actually started as 4 seperate projects independently. Firstly we had Morpheus v1 (the DBaaS offering we used for our companies), then we had Oohlalog. Oohlalog was our own log aggregation SaaS based product that we marketed publicly right around the time other big SaaS based logging products started showing up everywhere. It was great that we got to test these products with large amounts of data with the various portfolio companies of Bertram and work out any kinks with high load and big data we may need to work out before presenting as a public offering. Oohlalog was really an awesome logging tool and had great potential at the time, but there were too many other products that got out in the market at the time with stronger marketing and sales teams behind them.

Then there were two other SaaS based products done by two seperate teams independently within Bertram Labs. One was called Bitcan backups for automating and scheduling remote backups of various database types or file based backups without needing an agent. And another was called Happy Apps, an uptime monitoring tool similar to services like pingdom but much more in depth. It provides remote monitoring agents that could monitor not only websites but database services or message queues and even run custom db queries and present alerts based on the results. SSH connectivity was also allowed as well as paid customers could install their own happy apps agent on their own internal network that registered with happy apps SaaS to received monitoring tasks to run. Happy Apps and Bitcan are still both great products and growing on their own merits without much active marketing.

So we built all these cool SaaS products and realized something. We have a logging tool, we have a monitoring tool, we have a backup tool, and a database provisioning engine. What if we shifted the DBaaS to a full blown provisioning engine and orchestration tool, and combined all these other tools we built to provide one product that did everything you might need when managing a cloud? Better yet, we have a team intimately familiar with the challenges involved in all these types of tooling and how to scale it, secure it, and maintain it. The concept of Morpheus version 2 was born and when we started marketing it, received more momentum and excitement than any of the other products individually ever garnered. We were onto something.

When we rolled out Morpheus v2 for the first time, it was right around the time of the docker craze. Everyone was hyped about docker and so were we. We built the new version to run almost entirely on docker containers. We even made modifications such that our data was persisted rather than ephemeral within the context of a container as well as cross host network connectivity before docker even directly supported it. We wrote our own failover engine and scheduler that ensured the proper operation of all docker hosts in a cloud. It was and still is a very cool implementation of docker management. What we found though was , most companies were still new to the idea of Docker and unsure about it long term. They were not ready to necessarily commit to moving their engineering and IT team over to Docker based architecture for production workloads even though many large companies like Netflix were already doing so. We already knew how to automate provisioning of docker hosts on public clouds like Softlayer and Amazon so we decided to extend our provisioning abstraction to cover Virtual Machines and Baremetal servers.

Moving into VM based provisioning was a big project and at first was simple. When it first came out it was single disk, single network ,single security group and linux only. Now it has expanded to great detail on each cloud we support. The most in depth and complex integration would have to be VMWare followed by Amazon and Azure. VMware has a lot of fine grained control that our customers wanted to still provide in a self service portal. From picking target datastores in a multi disk environment, to multiple networks and task customizations. We support all of that now as well as reconfigure capabilities after the vm is already provisioned. One can even go to so much detail for vmware as selecting the network driver type or configuring the SCSI Controller (though these can be turned off for simplicity). We did all this, while still retaining an ease of install and ease of use.

There's so many integrations and features built on top of all of this and there are more features and flows to come as we keep up with a changing IT world that if I were to write about them now, we'd be talking about a novel. I plan to take more time to write about some of the challenges we faced as we have been growing as well as some of the use cases we've expanded to cover and use. Expect to see more posts about Morpheus as well as general development topics and plugin development topics.