I am on HRT as well. I had problems getting E2 dialed in correctly with Arimidex. I switched to Aromasin and things are much better. I wouldn't be surprised if your E2 is too low. That can make you feel like shit as well. Do your joints ache? If so, this could be a sign that your E2 is too low. Maybe try and cut back to .25 once every 3 days.
Good call. Some people are hypersensitive to adex, your e2 could be in the shitter rather than too high. I take 150mg test C/week and my estrogen bottoms out with anything more than .1mg adex EOD.. yes 0.1mg. Your symptoms sound like low e2 as well, especially if you aren't getting any bloating or nipple issues.
Try stopping the arimidex for a week, your e2 may blow up by the end of the week, but if you feel good at some point along the way(ie. passing through the sweet spot) then you'll know where you're at.
OR just get BW done haha