Hey scaredandworried, glad you are reaching out, and it sounds like you have the ability to check your emotions. Math is tough because you have to keep at it, or you can lose it. Calculus was tough for me, and then I went into applied combinatorics, which I don't even know if colleges teach this any more. Primarily for those studying Computer Science (which that was my major). Switched schools and majors by my junior year in college after I had a few mishaps in life. Graduated and got a good job, nevertheless.
Depression and anxiety can totally hijack your ability to do complex problems, and school, in general. It sounds like you have some self expectations that you really want to fulfill. Then, if you don't fulfill your expectations, you see yourself as a failure. If someone outside of your head spoke to you in the way that your own mind speaks to you, how would you respond to them?
Now, what I used to think ruined my life, and brought me to a place where I was willing to take my own life (got hospitalized etc.), has turned out to be the best thing that has ever happened to me. Life is not linear, but has many unexpected variations. That being the case, it is not predictable. One way to see how the mind cannot be trusted is to note how easy it is to believe (without a doubt) something awful will happen. Then how sluggish the mind is to embrace that life may be headed in a direction that is better than what we planned. It's hard for others to relate to the darkness that overcomes those of us with Major Depressive Disorder. Be very patient (even years patient). It is worth it to allow life to wash over you and make you into a unique human being that can help others. I'd suggest you keep up with school stuff (reduce to half time, for example), stay in the game, keep moving, and remember it is a long distance race that has a lot of struggle time in between the start and finish line. I kept moving, I never backed out of anything completely (even though I wanted to "take a break"). Try and fail. Fail as often as you want, just keep trying. Failure is okay. I do it all the time. You can still make it in life, even with a string of failures behind you. Many have failed. I feel sorry for the ones who are uncomfortable revealing that fact.