I recently encountered a situation where CI tests, which had been running fine on two systems for years, kept failing on the same test when we tried to run them on another system. It took us a few hours, but we eventually figured out the problem. As with most issues, it was multifactorial.

The Stage

• Most developers are programming on Macs, while production and CI run on Linux.
• There is a test that references a filepath similar to app/directory/Bestthing/file.csv.
• The test runs fine on developer machines and in CI.
• On developer machines, the file shows up as app/directory/BestThing/file.csv.
• Someone tried to utilize self-hosted CI runners that were using a Mac with a case-sensitive filesystem, and the test kept failing because the file was not found.
• While investigating, we discovered that the file was not found due to the case sensitivity of the filesystem. We updated the reference filepath to app/directory/BestThing/file.csv.
• The tests passed on the new self-hosted CI runners and local developer machines, but then they started failing in CI.
• We racked our brains trying to figure out why the file was now not found in CI.
• Eventually, I had the bright idea to click through the files on GitHub. What did I see?
• There was a folder called directory/Bestthing.

The Culprit

• Did you know you can somehow end up with a folder name that includes a /? I assume this might be a Linux filesystem quirk or a legacy issue. On Linux, it seemed to interpret this as a single folder, but on Mac, it interpreted it as two folders: directory and BestThing (yes, with a capital T) for some reason. This may have had to do with how Git was cloning and reproducing the file structure.
• We solved our issue by using GitHub Codespaces to load the repository, create the expected file structure, move the files to the correct locations, and then commit those changes.

Conclusion

Hopefully, this helps someone else out there who encounters a similar issue.