Engineering the Trading Platform: Inside Betterment’s Portfolio Optimization
To complete the portfolio optimization, Betterment engineers needed enhance the code in our existing trading platform. Here's how they did it.
In just a few weeks, Betterment is launching an updated portfolio -- one that has been optimized for better expected returns. The optimization will be partly driven by a more sophisticated asset allocation algorithm, which will dynamically vary individual asset allocations within the stock and bond basket based on a goal’s overall allocation.
This new flexible set of asset allocations significantly affects our current trading processes. Until now, we executed transactions based on fixed weights or a precise allocation of assets to every level of risk. Now, in our updated portfolio with a more sophisticated way to allocate, we are using a matrix to manage asset weights—and that requires more complex trading logic.
From an engineering perspective, this means we needed to enhance the code in our existing trading platform to accommodate dynamic asset allocation, with an eye towards future enhancements in our pipeline. Here's how we did it.
1. Build a killer testing framework
When dealing with legacy code, one of our top priorities is to preserve existing functionality. Failure to do so could mean anything from creating a minor inconvenience to blocking trades from executing. That means the next step was to build a killer testing framework. The novelty of our approach was to essentially build partial, precise scaffolding around our current platform.
This kind of scaffolding allowed us to go in and out of the current platform to capture and store precise inputs and outputs, while isolating them away from any unnecessary stuff that wasn’t relevant to the core trading processes.
2. Isolate the right information
With this abstraction, we were able to isolate the absolute core objects that we need to perform trades, and ignore the rest. This did two things: it took testing off the developers’ plates early in the process, allowing them to focus on writing production code, and also helped isolate the central objects that required most of their attention. The parent object of any activity inside the Betterment platform is a “user transaction” — that includes deposits or withdrawals to a goal, dividends, allocation changes, transfer of money between goals and so on.
The parent object of any activity inside the Betterment platform is a “user transaction” — that includes deposits or withdrawals for a goal, dividends, allocation changes, transfer of money between goals and so on.
These were our inputs. In most cases, a user transaction will eventually be the parent of several trade objects. These were our outputs. In our updated portfolio, the number of possible transactions types did not change. What did change, however, was how each transaction type was translated into trading activity, which is what we wanted to test exhaustively.
We captured a mass of user transaction objects from production for use in testing. However, a user transaction object contains a host of data that isn’t relevant to the trades that will eventually be created, and is associated with other objects that are also not relevant. So stripping out all non-trading data was the key to focusing on the right things to test for this project.
3. Use SQLite database to be efficient
The best way to store the user transaction objects was to use JSON, a human-readable representation of Java objects. To do this, we used GSON, which lets you convert Java objects into JSON, and vice versa. We didn’t want to store the JSON in a MySQL database, because managing it would be unnecessary overhead for this purpose. Instead, we stored them in a flat SQLite database. On the way into SQLite, GSON allowed us to “flatten” the objects, leaving only the bits that pertained to trading and discarding the rest.
Then, we could rearrange these chunks to replicate all sorts of trading activity patterns. On the way out, GSON would re-inflate the JSON back into Java objects, using dummy values for the irrelevant fields, providing us with test inputs ready to be pushed through our system. We did the same for outputs, which were also full of “noise” for our purposes. We’d shrink the expected results we got from production, then re-inflate and compare them to what our tests produced.
4. Do no harm to others' work
At Betterment, we are constantly pushing through new features and enhancements, some visible to customers, but many not. Development on these is concurrent, sometimes impacting global objects and schemas, and it was essential to insulate the team working on core trading functionality from all other development being done at the company.
Just the portfolio transition work alone includes significant new code for front-end enhancements which have nothing to do with trading. The GSON/JSON/SQLite testing framework helped the trading team maintain laser focus on their task, as they worked under the hood.
Otherwise, we’d be putting a sweet new set of tires on a car that won’t start!