Call me prejudiced but I am very reluctant to accept anything on the basis of a reference in any 'secret memoir': most of them are fictional. Probably the best source for Napoleon's personal peculiarities are Constant's memoirs, which can probably be accepted at this level.
Incidentally, Bill mentioned a story that Napoleon disliked men who were taller than himself (a large category) but I have just read in Marmont's memoirs that actually he was inclined to favour tall men and to promote them above their talents. Whitworth was probably using his height to look down
I have also just seen (this is coincidence I wasn't looking them up) a remark from Laure Junot about how much Napoleon hated the cold and had fires in his rooms from July - qv 'aversion to draughts'
The 'half a million men' remark was reported by Metternich and using it is likely to bring the supporters of Napoleon out rather than his detractors since they argue that nothing said by Napoleon's enemies (another large category) in his disfavour should be taken as truth. This limits the acceptable sources severely. The remark sounds to me exactly the sort of thing Napoleon might have said when showing-off though.
If trying to get inside the mind of Napoleon try reading his Correspondence - especially the unauthorised selections - you might decide it was somewhere you would rather not go!