Ginseng ficus: Gnarled bark or lots of scale?

Hi,
My poor Ginseng Ficus hasn't grown any new leaves in over 2 years and occasionally drops a batch of yellow leaves. Being the beginner I am, I always thought it was something about the conditions (temperature, watering, lack of fertilizer / light), but none of that has really helped. It's still got a few nice green leaves, but it's quite bare.

At one point it had spider mites, which I treated successfully years ago and this doesn't look like that. I also didn't think of scale at first because my girlfriend's bonsai definitely had that (in another apartment, last year) and we picked off those buggers fairly easily.

The gnarliness on my tree's trunk doesn't look like that, but after I accidentally knocked some of this "stuff" off, the marks left behind reminded me of scale bugs. If it IS scale or something like it, I suppose that would explain the lack of growth because it's just struggling to stay alive.

Could folks take a look and see if I'm right, and if not, is this just what the bark is supposed to look like? I'm not certain this is normal because in some places it DOES appear quite smooth better-looking.

Thanks,
Rafael