My yield curve site yields.paulsulli.com has continued to be convenient this year with all the interest rate hikes we’ve seen. I check it myself and have a few friends that use it too. But one day a month or so ago one of them complained that the site wasn’t loading. That’s weird, I didn’t change anything. Well it turned out that the US Treasury, which I’m using as the source of the data, did make an update. In addition to changing their formatting slightly, they seem to have introduced a ~10 second delay in downloading files. I suspect this is some sort of rate limiting. The problem is that when I slapped this together I simply had code in an AWS Lambda that would run every time the site is loaded and query the Treasury to get data for this year and the previous two years. Because of the new rate limiting it now takes ~30 seconds for the page to load. Not good!

Of course, pulling the data every time the site loads wasn’t a great design to begin with. But the volume is low enough that it didn’t seem like that big of a deal. A more efficient approach is to only query this year’s data once a day to get the latest data, then store it somewhere for the lambda that displays my page to use. So I made a second lambda that queries the Treasury site once a day after the data is updated and stores it in S3. I also added a couple of files for previous years, but those don’t need to be updated because the data is already all there. I then modified the display lambda to get these files from S3 and process them as usual. And it worked. The page now loads basically instantly.