You shouldn’t do that after import, as it initializes the model, i.e. it overwrites your imported weights.
But that isn’t the problem yet, as it does’t even get that far.
Because what you’re seeing is a validation error, I guess there is not way around defining an input shape.
But I think you should still be able to apply the imported model to arbitrary sized images (as long as they are big enough to not run into size issues at another point).