Question linkage Using model predictions with multiple threads can cause results to be overwritten。
I tried to use the model prediction under multi-threaded and found that some results were overwritten, then I tested and found that it seems that the results of loader.asMatrix are overwritten under multi-threaded, he is not thread safe right, how can I load the image under multi-threaded.
I was using NativeImageLoader and it was messing up with multiple threads, but then I tried using ImageLoader and I don’t have a problem. I don’t know if this is a solution.
@visionki could you clarify? Is the input data overridden? Models themselves are not thread safe. You need to use parallelinference.
@agibsonccc Yes, I used parallelinference, Here is my code snippet:
@Slf4j
@Component
public class NsfwUtil {
ComputationGraph model;
private final ParallelInference parallelInference;
private final ImageLoader imgLoader;
NativeImageLoader loader;
public NsfwUtil() throws IOException, UnsupportedKerasConfigurationException, InvalidKerasConfigurationException {
model = KerasModelImport.importKerasModelAndWeights(new ClassPathResource("model/nsfw.299x299.h5").getInputStream());
parallelInference = new ParallelInference.Builder(model)
.inferenceMode(InferenceMode.BATCHED)
.batchLimit(32)
.workers(5)
.build();
loader = new NativeImageLoader(299, 299, 3, new ColorConversionTransform(COLOR_BGR2RGB));
imgLoader = new ImageLoader(299, 299, 3);
}
@Async("loadExecutor")
public CompletableFuture<AnalysisResult> getResultAsync(String url){
try (InputStream inputStream = Unirest.get(url)
.asObjectAsync(RawResponse::getContent)
.get()
.getBody()){
INDArray matrix = loader.asMatrix(inputStream, false);
log.info("{}", matrix.hashCode());
INDArray output = parallelInference.output(matrix.div(255));
return CompletableFuture.completedFuture(new AnalysisResult(output.data().asDouble()));
}catch (Exception e){
throw new RuntimeException();
}
}
}
The above example uses NativeImageLoader to load images, and the output of matrix.hashCode() is occasionally duplicated (common when first started), and the log is as follows:
735038670
735038670
988874855
988874855
2103345588
-822422519
920533466
-1450966807
-1952245710
-2122104256
-82726789
When I replace it with ImageLoader to load the image, this does not happen.
@visionki I would recommend a threadlocal for the NativeImageLoader. That should help your issue.
@agibsonccc I want to know what is the difference between NativeImageLoader and ImageLoader, I can’t find the relevant documentation. looking at the comments it seems NativeImageLoader is to load images with opencv, ImageLoader is to load images with BufferedImage, do you mean you don’t recommend using ImageLoader?
@visionki NativeImageLoader uses opencv and is the recommended way of loading images. There’s not really much to document there.
@agibsonccc Ok, thanks for your reply, I’ll try what you said.
@agibsonccc Hello, still similar code as above, I found that after running for a while the memory footprint will keep increasing, memory footprint is:
When I periodically execute system.gc() manually, the memory footprint seems to become normal:
What do I need to release so that memory doesn’t keep growing?
@visionki that’s pretty common with threadlocals if not managed properly. Are you creating new threads every time?
