Cannot access Webserver

Hi there,

I am running the example MultiDigitNumberRecognition and tried accessing the UI.
In order to do so I needed to change the port, wich worked well.
Anyhow, I am not able to access the webserver.

I checked for correct port, address and status during runtime. All seem to be well. For the status I used the method isStopped(). I got false back.

Someone got a clue what I’m doing wrong?

BR
chris

Could you give us more details? How are you running it exactly? On a local laptop? A remote server?

I’m running the example on a PC through CPU (i7-3770).
It’s the DL4J-Version Beta 7. The OS is Windows 10. The IDE I’m using is Eclipse and the JRE 1.8.0_261.

I read that the Beta 6 got some issues with the UI, so I am guessing Beta 7 should be fine. I didn’t try Beta 5 yet.

@chris are you just running it from intellij then? If you are, I’ll take a look. That should work fine out of the box. Do you have any stack traces I can look at?

I didn’t try intellij yet. I will check.

I don’t have a stack trace yet, since I got no error. But I could set a break point and check for valid variable content.

@agibsonccc, I installed intellij and got the example running, but still cannot access the webserver. I also didn’t got any error. The example itself runs well. You have any idea what might be wrong? Can I check something else?

@chris that’s why I asked how you were running it. I’m actually still not sure what your issue is you haven’t told me how you’re running it. I’m guessing you’re trying to use nginx or something. Sorry you haven’t given me a stack trace or anything.

You can’t expect me to read your computer or your mind or reverse engineer your circumstances from just a few sentences. Try to help me out a bit :slight_smile:

Try to give me an end to end example like “I’m running with an uber jar on a local server and starting it with this command. Here’s my logs that show the issue” or something like that.

I’m just running it out of the box (the port is changed from 9000 to 9001 through via environment parameter), I got the example also runinning on port 9000 (closed the other app using it first)

The example is using the vertx server as I saw. How I get the stack trace in intellij, just break at a reasonable point in the server? :slightly_smiling_face:

The following is the output I get from the example:

“C:\Program Files\Java\jdk1.8.0_261\bin\java.exe” -Dorg.deeplearning4j.ui.port=9001 “-javaagent:C:\Program Files\JetBrains\IntelliJ IDEA Community Edition 2020.2.2\lib\idea_rt.jar=55619:C:\Program Files\JetBrains\IntelliJ IDEA Community Edition 2020.2.2\bin” -Dfile.encoding=UTF-8 -classpath “C:\Program Files\JetBrains\IntelliJ IDEA Community Edition 2020.2.2\lib\idea_rt.jar” com.intellij.rt.execution.CommandLineWrapper C:\Users\chris\AppData\Local\Temp\idea_classpath524300165 org.deeplearning4j.examples.advanced.modelling.captcharecognition.MultiDigitNumberRecognition
1601377066503
o.d.e.a.m.c.MultiDigitNumberRecognition - C:\Users\chris\Documents\workspaces\out\models\validateCodeCheckModel.json
o.n.l.f.Nd4jBackend - Loaded [CpuBackend] backend
o.n.n.NativeOpsHolder - Number of threads used for linear algebra: 4
o.n.l.c.n.CpuNDArrayFactory - *********************************** CPU Feature Check Warning ***********************************
o.n.l.c.n.CpuNDArrayFactory - Warning: Initializing ND4J with Generic x86 binary on a CPU with AVX/AVX2 support
o.n.l.c.n.CpuNDArrayFactory - Using ND4J with AVX/AVX2 will improve performance. See deeplearning4j.org/cpu for more details
o.n.l.c.n.CpuNDArrayFactory - Or set environment variable ND4J_IGNORE_AVX=true to suppress this warning
o.n.l.c.n.CpuNDArrayFactory - *************************************************************************************************
o.n.n.Nd4jBlas - Number of threads used for OpenMP BLAS: 4
o.n.l.a.o.e.DefaultOpExecutioner - Backend used: [CPU]; OS: [Windows 10]
o.n.l.a.o.e.DefaultOpExecutioner - Cores: [8]; Memory: [3,5GB];
o.n.l.a.o.e.DefaultOpExecutioner - Blas vendor: [OPENBLAS]
o.d.n.g.ComputationGraph - Starting ComputationGraph with WorkspaceModes set to [training: ENABLED; inference: ENABLED], cacheMode set to [NONE]
o.d.u.VertxUIServer - Deeplearning4j UI server started at: http://localhost:9001
Port: 9001
o.d.u.VertxUIServer - StatsStorage instance attached to UI: FileStatsStorage(C:\Users\TSCHEN~1\AppData\Local\Temp\ui-stats.dl4j)


Example data present in
C:\Users\chris\dl4j-examples-data\dl4j-examples\captchaImage



Example data present in
C:\Users\chris\dl4j-examples-data\dl4j-examples\captchaImage



Example data present in
C:\Users\chris\dl4j-examples-data\dl4j-examples\captchaImage


@agibsonccc, I figured out in itellij (by thread dump) that a vertx thread is blocked.

"vertx-blocked-thread-checker@1965" daemon prio=5 tid=0x12 nid=NA waiting
  java.lang.Thread.State: WAITING
	  at java.lang.Object.wait(Object.java:-1)
	  at java.util.TimerThread.mainLoop(Timer.java:552)
	  at java.util.TimerThread.run(Timer.java:505)

Here the complete dump:

"main@1" prio=5 tid=0x1 nid=NA runnable
  java.lang.Thread.State: RUNNABLE
	  at org.nd4j.nativeblas.Nd4jCpu.execCustomOp2(Nd4jCpu.java:-1)
	  at org.nd4j.linalg.cpu.nativecpu.ops.NativeOpExecutioner.exec(NativeOpExecutioner.java:1921)
	  at org.nd4j.linalg.cpu.nativecpu.ops.NativeOpExecutioner.exec(NativeOpExecutioner.java:1573)
	  at org.nd4j.linalg.factory.Nd4j.exec(Nd4j.java:6566)
	  at org.deeplearning4j.ui.model.stats.BaseStatsListener.getHistograms(BaseStatsListener.java:769)
	  at org.deeplearning4j.ui.model.stats.BaseStatsListener.iterationDone(BaseStatsListener.java:473)
	  at org.deeplearning4j.optimize.solvers.StochasticGradientDescent.optimize(StochasticGradientDescent.java:105)
	  at org.deeplearning4j.optimize.Solver.optimize(Solver.java:52)
	  at org.deeplearning4j.nn.graph.ComputationGraph.fitHelper(ComputationGraph.java:1166)
	  - locked <0x1099> (a org.deeplearning4j.nn.graph.ComputationGraph)
	  at org.deeplearning4j.nn.graph.ComputationGraph.fit(ComputationGraph.java:1116)
	  at org.deeplearning4j.nn.graph.ComputationGraph.fit(ComputationGraph.java:1083)
	  at org.deeplearning4j.nn.graph.ComputationGraph.fit(ComputationGraph.java:1045)
	  at org.deeplearning4j.examples.advanced.modelling.captcharecognition.MultiDigitNumberRecognition.main(MultiDigitNumberRecognition.java:97)
	  at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1)
	  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	  at java.lang.reflect.Method.invoke(Method.java:498)
	  at com.intellij.rt.execution.CommandLineWrapper.main(CommandLineWrapper.java:63)

"vert.x-eventloop-thread-1@2886" prio=5 tid=0x19 nid=NA runnable
  java.lang.Thread.State: RUNNABLE
	  at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(WindowsSelectorImpl.java:-1)
	  at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
	  at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
	  at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
	  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	  - locked <0x108a> (a sun.nio.ch.WindowsSelectorImpl)
	  - locked <0x108b> (a java.util.Collections$UnmodifiableSet)
	  - locked <0x108c> (a io.netty.channel.nio.SelectedSelectionKeySet)
	  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
	  at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68)
	  at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:803)
	  at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:457)
	  at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	  at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	  at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	  at java.lang.Thread.run(Thread.java:748)

"vert.x-acceptor-thread-0@2870" prio=5 tid=0x18 nid=NA runnable
  java.lang.Thread.State: RUNNABLE
	  at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(WindowsSelectorImpl.java:-1)
	  at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
	  at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
	  at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
	  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	  - locked <0x108d> (a sun.nio.ch.WindowsSelectorImpl)
	  - locked <0x108e> (a java.util.Collections$UnmodifiableSet)
	  - locked <0x108f> (a io.netty.channel.nio.SelectedSelectionKeySet)
	  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
	  at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68)
	  at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:803)
	  at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:457)
	  at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	  at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	  at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	  at java.lang.Thread.run(Thread.java:748)

"vert.x-eventloop-thread-0@2367" prio=5 tid=0x15 nid=NA runnable
  java.lang.Thread.State: RUNNABLE
	  at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(WindowsSelectorImpl.java:-1)
	  at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
	  at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
	  at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
	  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	  - locked <0x1090> (a sun.nio.ch.WindowsSelectorImpl)
	  - locked <0x1091> (a java.util.Collections$UnmodifiableSet)
	  - locked <0x1092> (a io.netty.channel.nio.SelectedSelectionKeySet)
	  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
	  at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68)
	  at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:803)
	  at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:457)
	  at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	  at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	  at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	  at java.lang.Thread.run(Thread.java:748)

"Thread-7@2560" daemon prio=5 tid=0x17 nid=NA waiting
  java.lang.Thread.State: WAITING
	  at sun.misc.Unsafe.park(Unsafe.java:-1)
	  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	  at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	  at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	  at org.deeplearning4j.ui.VertxUIServer$StatsEventRouterRunnable.runHelper(VertxUIServer.java:582)
	  at org.deeplearning4j.ui.VertxUIServer$StatsEventRouterRunnable.run(VertxUIServer.java:570)
	  at java.lang.Thread.run(Thread.java:748)

"Thread-6@2861" prio=5 tid=0x16 nid=NA waiting
  java.lang.Thread.State: WAITING
	  at java.lang.Object.wait(Object.java:-1)
	  at java.lang.Thread.join(Thread.java:1252)
	  at java.lang.Thread.join(Thread.java:1326)
	  at org.deeplearning4j.ui.VertxUIServer.lambda$deploy$8(VertxUIServer.java:194)
	  at org.deeplearning4j.ui.VertxUIServer$$Lambda$15.51376124.run(Unknown Source:-1)
	  at java.lang.Thread.run(Thread.java:748)

"AMDSI prefetch thread@3610" daemon prio=5 tid=0x1b nid=NA waiting
  java.lang.Thread.State: WAITING
	  at sun.misc.Unsafe.park(Unsafe.java:-1)
	  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	  at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	  at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350)
	  at org.nd4j.linalg.dataset.AsyncMultiDataSetIterator$AsyncPrefetchThread.run(AsyncMultiDataSetIterator.java:377)

"DeallocatorServiceThread_1@1627" daemon prio=5 tid=0x11 nid=NA waiting
  java.lang.Thread.State: WAITING
	  at java.lang.Object.wait(Object.java:-1)
	  at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
	  at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
	  at org.nd4j.linalg.api.memory.deallocation.DeallocatorService$DeallocatorServiceThread.run(DeallocatorService.java:129)

"DeallocatorServiceThread_0@1626" daemon prio=5 tid=0x10 nid=NA waiting
  java.lang.Thread.State: WAITING
	  at java.lang.Object.wait(Object.java:-1)
	  at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
	  at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
	  at org.nd4j.linalg.api.memory.deallocation.DeallocatorService$DeallocatorServiceThread.run(DeallocatorService.java:129)

"JavaCPP Deallocator@1430" daemon prio=10 tid=0xf nid=NA waiting
  java.lang.Thread.State: WAITING
	  at java.lang.Object.wait(Object.java:-1)
	  at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
	  at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
	  at org.bytedeco.javacpp.Pointer$DeallocatorThread.run(Pointer.java:375)

"Thread-5@2269" daemon prio=5 tid=0x14 nid=NA runnable
  java.lang.Thread.State: RUNNABLE
	  at sun.net.dns.ResolverConfigurationImpl.notifyAddrChange0(ResolverConfigurationImpl.java:-1)
	  at sun.net.dns.ResolverConfigurationImpl$AddressChangeListener.run(ResolverConfigurationImpl.java:144)

"vertx-blocked-thread-checker@1965" daemon prio=5 tid=0x12 nid=NA waiting
  java.lang.Thread.State: WAITING
	  at java.lang.Object.wait(Object.java:-1)
	  at java.util.TimerThread.mainLoop(Timer.java:552)
	  at java.util.TimerThread.run(Timer.java:505)

"Finalizer@4232" daemon prio=8 tid=0x3 nid=NA waiting
  java.lang.Thread.State: WAITING
	  at java.lang.Object.wait(Object.java:-1)
	  at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
	  at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
	  at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)

"Reference Handler@4233" daemon prio=10 tid=0x2 nid=NA waiting
  java.lang.Thread.State: WAITING
	  at java.lang.Object.wait(Object.java:-1)
	  at java.lang.Object.wait(Object.java:502)
	  at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
	  at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

"Attach Listener@4230" daemon prio=5 tid=0x5 nid=NA runnable
  java.lang.Thread.State: RUNNABLE

"Signal Dispatcher@4231" daemon prio=9 tid=0x4 nid=NA runnable
  java.lang.Thread.State: RUNNABLE

@agibsonccc, I figured out by the intellij thread dump that a vertx thread seems to be blocked.

"vertx-blocked-thread-checker@1965" daemon prio=5 tid=0x12 nid=NA waiting
  java.lang.Thread.State: WAITING
	  at java.lang.Object.wait(Object.java:-1)
	  at java.util.TimerThread.mainLoop(Timer.java:552)
	  at java.util.TimerThread.run(Timer.java:505)

@agibsonccc, I got also another finding. The server address is 0.0.0.0 instead of localhost. So it seems to be a configuration failure of the UI server.

The meta address isn’t the issue. I can establish a stable TCP connection to the server, but without getting any response on a GET request.

I checked also the vertx server. I can trigger an internal call from the browser. So it seems to be something wrong the listener implementation. I also tried the entire setup on another PC with intellij and eclipse.
So currently with Beta 7 the UI access won’t work (with Windows 10).

Is someone familiar with the vertx implementation and might be able to fix it?