See all articles

Dealing With the Uncertainty of Legacy Code

To complete our portfolio optimization, we had to tackle a lot of legacy code. And then we applied our learnings going forward.

By Greg Sadowski

Staff Engineer, Betterment |

Published Mar. 31, 2014

Dealing With the Uncertainty of Legacy Code Mar 31, 2014 12:00:00 AM To complete our portfolio optimization, we had to tackle a lot of legacy code. And then we applied our learnings going forward. Last fall, Betterment optimized its portfolio, moving from the original platform to an upgraded trading platform that included more asset classes and the ability to weight exposure of each asset class differently for every level of risk. For Betterment engineers, it meant restructuring the underlying portfolio data model for increased flexibility. For our customers, it should result in better expected, risk-adjusted returns for investments. However, as our data model changed, pieces of the trading system also had to change to account for the new structure. While most of this transition was smooth, there were a few cases where legacy code slowed our progress. To be sure, we don't take changing our system lightly. While we want to iterate rapidly, we strive to never compromise the security of our customers nor the correctness of our code. For this reason, we have a robust testing infrastructure and only peer-reviewed, thoroughly-tested code gets pushed through to production. What is legacy code? While there are plenty of metaphors and ways to define legacy code, it has this common feature: It’s always tricky to work with it. The biggest problem is that sometimes you're not always sure the original purpose of older code. Either the code is poorly designed, the code has no tests around it to specify its behavior, or both. Uncertainty like this makes it hard to build new and awesome features into a product. Engineers' productivity and happiness decrease as even the smallest tasks can be frustrating and time-consuming. Thus, it’s important for engineers to do two things well: (a) be able to remove existing legacy code and (b) not to write code that is likely to become legacy code in the future. Legacy code is a form of technical debt—the sooner it gets fixed, the less time it will take to fix in the future. How to remove legacy code During our portfolio optimization, we had to come up with a framework for dealing with pieces of old code. Here’s what we considered: We made sure we knew its purpose. If the code is not on any active or planned future development paths and has been working for years, it probably isn't worth it. Legacy code can take a long time to properly test and remove. We made a good effort to understand it. We talked to other developers who might be more familiar with it. During the portfolio update project, we routinely brought a few engineers together to diagram trading system flow on a whiteboard. We wrote tests around the methods in question. It's important to have tests in place before changing code to be as confident as possible that the behavior of the code is not changing during refactoring. Hopefully, it is possible to write unit tests for at least a part of the method's behavior. Write unit tests for a piece of the method, then refactor that piece. Test, repeat, test. Once the tests are passing, write more tests for the next piece, and repeat the test, refactor, test, refactor process. Fortunately, we were able to get rid of most of the legacy code encountered during the portfolio optimization project using this method. Then there are outliers Yet sometimes even the best practices still didn’t apply to a piece of legacy code. In fact, sometimes it was hard to even know where to start to make changes. In my experience, the best approach was to jump in and rewrite a small piece of code that was not tested, and then add tests for the rewritten portion appropriately. Write characterization tests We also experimented with characterization tests. First proposed by Michael Feathers (who wrote the bible on working with legacy code) these tests simply take a set of verified inputs/outputs from the existing production legacy code and then assert that the output of the new code is the same as the legacy code under the same inputs. Several times we ran into corner cases around old users, test users, and other anomalous data that caused false positive failures in our characterization tests. These in turn led to lengthy investigations that consumed a lot of valuable development time. For this reason, if you do write characterization tests, we recommend not going too far with them. Handle a few basic cases and be done with them. Get better unit or integration tests in place as soon as possible. Build extra time into project estimates Legacy code can also be tricky when it comes to project estimates. It is notoriously hard to estimate the complexity of a task when it needs to be built into or on top of a legacy system. In our experience, it has always taken longer than expected. The portfolio optimization project took longer than initially estimated. Also, if database changes are part of the project (e.g. dropping a database column that no longer makes sense in the current code structure), it's safe to assume that there will be data issues that will consume a significant portion of developer time, especially with older data. Apply the learnings to future The less legacy code we have, the less we have to deal with the aforementioned processes. The best way to avoid legacy code is to make a best effort at not writing in the first place. The best way to avoid legacy code is to make a best effort at not writing it in the first place. For example, we follow a set of pragmatic design principles drawn from SOLID (also created by Michael Feathers) to help ensure code quality. All code is peer reviewed and does not go to production if there is not adequate test coverage or if the code is not up to design standards. Our unit tests are not only to test behavior and drive good design, but should also be readable to the extent that they help document the code itself. When writing code, we try to keep in mind that we probably won't come back later and clean up the code, and that we never know who the next person to touch this code will be. Betterment has also established a "debt day" where once every month or two, all developers take one day to pay down technical debt, including legacy code. The Results It's important to take a pragmatic approach to refactoring legacy code. Taking the time to understand the code and write tests before refactoring will save you headaches in the future. Companies should strive for a fair balance between adding new features and refactoring legacy code, and should establish a culture where thoughtful code design is a priority. By incorporating many of these practices, it is steadily becoming more and more fun to develop on the Betterment platform. And the Betterment engineering team is avoiding the dreaded productivity and happiness suck that happens when working on systems with too much legacy code. Interested in engineering at Betterment? Betterment is an engineering-driven company that has developed the most-trusted online financial advisor based on the principles of optimization and efficiency. Learn more about engineering jobs and our culture. Determination of most trusted online financial advisor reflects Betterment LLC's distinction of having the most customers in the industry, made in reliance on customer counts, self-reported pursuant to SEC rules, across all online-only registered investment advisors.

Last fall, Betterment optimized its portfolio, moving from the original platform to an upgraded trading platform that included more asset classes and the ability to weight exposure of each asset class differently for every level of risk.

For Betterment engineers, it meant restructuring the underlying portfolio data model for increased flexibility. For our customers, it should result in better expected, risk-adjusted returns for investments.

However, as our data model changed, pieces of the trading system also had to change to account for the new structure. While most of this transition was smooth, there were a few cases where legacy code slowed our progress.

To be sure, we don't take changing our system lightly. While we want to iterate rapidly, we strive to never compromise the security of our customers nor the correctness of our code. For this reason, we have a robust testing infrastructure and only peer-reviewed, thoroughly-tested code gets pushed through to production.

What is legacy code?

While there are plenty of metaphors and ways to define legacy code, it has this common feature: It’s always tricky to work with it. The biggest problem is that sometimes you're not always sure the original purpose of older code. Either the code is poorly designed, the code has no tests around it to specify its behavior, or both.

Uncertainty like this makes it hard to build new and awesome features into a product. Engineers' productivity and happiness decrease as even the smallest tasks can be frustrating and time-consuming. Thus, it’s important for engineers to do two things well: (a) be able to remove existing legacy code and (b) not to write code that is likely to become legacy code in the future.

Legacy code is a form of technical debt—the sooner it gets fixed, the less time it will take to fix in the future.

How to remove legacy code

During our portfolio optimization, we had to come up with a framework for dealing with pieces of old code. Here’s what we considered:

We made sure we knew its purpose. If the code is not on any active or planned future development paths and has been working for years, it probably isn't worth it. Legacy code can take a long time to properly test and remove.
We made a good effort to understand it. We talked to other developers who might be more familiar with it. During the portfolio update project, we routinely brought a few engineers together to diagram trading system flow on a whiteboard.
We wrote tests around the methods in question. It's important to have tests in place before changing code to be as confident as possible that the behavior of the code is not changing during refactoring. Hopefully, it is possible to write unit tests for at least a part of the method's behavior. Write unit tests for a piece of the method, then refactor that piece.
Test, repeat, test. Once the tests are passing, write more tests for the next piece, and repeat the test, refactor, test, refactor process. Fortunately, we were able to get rid of most of the legacy code encountered during the portfolio optimization project using this method.

Then there are outliers

Yet sometimes even the best practices still didn’t apply to a piece of legacy code. In fact, sometimes it was hard to even know where to start to make changes. In my experience, the best approach was to jump in and rewrite a small piece of code that was not tested, and then add tests for the rewritten portion appropriately.

Write characterization tests

We also experimented with characterization tests. First proposed by Michael Feathers (who wrote the bible on working with legacy code) these tests simply take a set of verified inputs/outputs from the existing production legacy code and then assert that the output of the new code is the same as the legacy code under the same inputs.

Several times we ran into corner cases around old users, test users, and other anomalous data that caused false positive failures in our characterization tests. These in turn led to lengthy investigations that consumed a lot of valuable development time.

For this reason, if you do write characterization tests, we recommend not going too far with them. Handle a few basic cases and be done with them. Get better unit or integration tests in place as soon as possible.

Build extra time into project estimates

Legacy code can also be tricky when it comes to project estimates. It is notoriously hard to estimate the complexity of a task when it needs to be built into or on top of a legacy system.

In our experience, it has always taken longer than expected. The portfolio optimization project took longer than initially estimated. Also, if database changes are part of the project (e.g. dropping a database column that no longer makes sense in the current code structure), it's safe to assume that there will be data issues that will consume a significant portion of developer time, especially with older data.

Apply the learnings to future

The less legacy code we have, the less we have to deal with the aforementioned processes. The best way to avoid legacy code is to make a best effort at not writing in the first place. The best way to avoid legacy code is to make a best effort at not writing it in the first place.

For example, we follow a set of pragmatic design principles drawn from SOLID (also created by Michael Feathers) to help ensure code quality. All code is peer reviewed and does not go to production if there is not adequate test coverage or if the code is not up to design standards. Our unit tests are not only to test behavior and drive good design, but should also be readable to the extent that they help document the code itself. When writing code, we try to keep in mind that we probably won't come back later and clean up the code, and that we never know who the next person to touch this code will be. Betterment has also established a "debt day" where once every month or two, all developers take one day to pay down technical debt, including legacy code.

The Results

It's important to take a pragmatic approach to refactoring legacy code. Taking the time to understand the code and write tests before refactoring will save you headaches in the future. Companies should strive for a fair balance between adding new features and refactoring legacy code, and should establish a culture where thoughtful code design is a priority. By incorporating many of these practices, it is steadily becoming more and more fun to develop on the Betterment platform. And the Betterment engineering team is avoiding the dreaded productivity and happiness suck that happens when working on systems with too much legacy code.

Interested in engineering at Betterment? Betterment is an engineering-driven company that has developed the most-trusted online financial advisor based on the principles of optimization and efficiency.

Learn more about engineering jobs and our culture.

Determination of most trusted online financial advisor reflects Betterment LLC's distinction of having the most customers in the industry, made in reliance on customer counts, self-reported pursuant to SEC rules, across all online-only registered investment advisors.