We recently moved one of our C2 Labs applications to use the new 3.1 version of the .NET Core Framework. While the move from version 2 to version 3 was not without pain, we were excited to be on the latest version and to start taking advantage of its new features. We were able to re-factor our code, test locally, and get everything working fairly quickly. However, we ran into a weird issue once we deployed.
When testing locally, all of the features worked. However, we deploy via Docker containers through our CI/CD into a Kubernetes (K8s) cluster in Azure. Once the code was running in a container, everything worked except for one feature. We were no longer able to send email via SMTP using the .NET Core System.Net.Mail library. Every email bombed and threw errors when running in the K8s cluster. However, everything worked perfectly when compiling and running in debug on our localhost.
This error ended up taking about half a day to troubleshoot and fix. Like most errors of this type, it is hard to diagnose when you cannot reproduce the error locally. We thought we would share the fix with others to hopefully save you guys the time and effort it took us to resolve.
Let's start with the error we were seeing in the container:
System.BadImageFormatException: [/app/System.Net.Mail.dll] Invalid Typeref token.We were using the exact same code base and version of .NET Core across localhost and the K8s environment so we knew that couldn't be the issue. The only thing that was different was that the deployed application was running in a container. Turns out the issue was in the Dockerfile on this line:
RUN dotnet publish -c release -o /app -r linux-x64 --self-contained true --no-restore /p:PublishTrimmed=true /p:PublishReadyToRun=trueIn particular, it was the PublishTrimmed feature that was causing the issue. This is one of those new .NET core features we were hoping to take advantage of. The goal was that it would trim out unused libraries from the framework and create a smaller compiled executable that gets deployed in the container. However, it does not appear to work in our case. It ended up stripping out the System.Net.Mail library which broke the ability to send code from the container. Consequently, we had to rollback this feature by updating the Dockerfile publish command as follows:
RUN dotnet publish -c release -o /app -r linux-x64 --self-contained true --no-restoreBy removing new publish commands, all libraries are included in the container again and the code is back in a working state. While this results in some inefficiency (a bigger compiled executable), it also results in working code which is kind of the whole idea. :)
When troubleshooting this issue, it turns out there isn't a lot of good documentation out there since this version of .NET Core is so new. We hope this blog helps explain the issue for others and saves you some time along the way.
Interested in containerizing and modernizing your applications? Contact us today for a free one hour consultation to find out how our C2 Labs DevOps engineers can accelerate your digital transformation.