I made it faster by surrounding the world with border voxels, which allows for the edge of world check to happen once per ray, not for each voxel in it's path. I also made it calculate the voxel index once, and update it during the loop rather than recalculating it. This could still be made faster, but such optimizations would be tricky to implement.