I agree that he must have been on good terms with the Bourbon government, but don't see how you know he served under the Bourbons. Retired officers are generally allowed to wear their uniforms. Are you making the implication that if he was decorated by the Bourbons then he must have served them? I can see the logic in that, but don't know that it is correct.
As for serving under the Empire, how do you come to that conclusion? Didn't the Bourbons also give the Legion d'honeur?
Lots of people held rank in armies in which they never served, and had portraits made in that uniform. They were normally royalty or highly respected, such as Wellington, but it was a mark of respect - sort of like an honourary degree from a university.