How did you prepare your inoculum and what was the number of rhizobia per seed you added?
How did your surface sterilize the seeds
usually agar is not the best media to test nodulation and nitrogen fixation.
Now in a population of rhizobia, you have some which are very efficient, and some which are not efficient at all, in this later case the bacteria are acting as minor pathogens and yes you can have a yield lower than the uninoculated control. They form ineffective white nodules.
So you need to compare the nodules, size color etc...in order to appreciate efficiency.   _{Your Comment}