Hello, We recently started seeing repeated [web] Internal Server Error warnings in our Application Director of our Niagara 3.8.11 installation. The warnings happen about twice a minute and look like this:

ERROR [12:49:27 05-Jun-17 PDT][web] Internal Server Error
java.lang.NullPointerException
  at com.tridium.web.WebProcess.handlePasswordReset(WebProcess.java:457)
  at com.tridium.web.WebProcess.challenge(WebProcess.java:401)
  at com.tridium.web.WebProcess.service(WebProcess.java:118)
  at com.tridium.web.SysServlet.service(SysServlet.java:114)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:805)
  at com.tridium.httpd.ServiceThread.handleRequest(ServiceThread.java:356)
  at com.tridium.httpd.ServiceThread.run(ServiceThread.java:109)

These errors seem to coincide with nHaystack, as we've determined that we can make the errors stop by disabling the connectors in all our SkySpark projects that use Haystack.

Ever since we started seeing these Internal Server errors, nHaystack performance has slowed down considerably, and memory usage on the Niagara computer has gone up considerably.

Has anyone seen this error and/or behavior? We running on a conventional Windows 10 environment, and everything ran smoothly for several years until last week.

Thanks, Jerry

Hi Jerry,

I haven't had anyone else report this error. Is there anything special about your installation on this site?

When you say you recently saw these warnings what do you mean? What else happend on this installation around the time these warnings started to appear?

Cheers, Richard

Richard, OK, here's what we've been able to discern so far. We initially had nHaystack running on an AX 3.7 installation. This problem seems to have started when AX 3.8 was installed on the computer, although we hadn't started using 3.8 yet. In both 3.7, and then 3.8 when we migrated over, we received the error above, which after a large number of time would seem to be condensed and logged as just:

ERROR [07:07:29 06-Jun-17 PDT][web] Internal Server Error
java.lang.NullPointerException

We ultimately changed the password for the Tridium user accounts and updated the password for all the SkySpark Haystack connectors, and that seemed to have resolved the passwordReset error. But we still get the above NullPointerException error, recurring when SkySpark projects sync histories through their Haystack connectors. The history syncs are now much faster and successful, so hopefully the error above is now just console spam.

Can you provide "spy" logs from the web service ? Maybe there is more informations in there...

I assume you've restarted the station a few times?

Did you try rolling back to 3.7?

Cheers, Richard

Yes, we've restarted the station several times while debugging. We have not rolled back to 3.7.

Where do we get the spy logs? I've retrieved the Remote Spy's WebServer proerties, but it looks more like a configuration page than a log. We've also set the Remote Spy's settings on the web server from Message to Trace.

Jerry

We've set trace logging on the web service and all web. related services, and have a bit more detailed information in our application manager logging. I can provide an export of the logs, but these three errors are the most proficient by far:

• [web] Internal Server Error: java.lang.NullPointerException
• [web] Internal Server Error: java.net.SocketException: Software caused connection abort: socket write error

  at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109) at java.net.SocketOutputStream.write(SocketOutputStream.java:153) at com.tridium.net.ChunkedOutputStream.writeChunk(ChunkedOutputStream.java:82)

• [web.service] HttpdThread-473: Error servicing request: java.lang.IllegalStateException

  at com.tridium.httpd.HttpResponse.sendError(HttpResponse.java:431) at com.tridium.web.SysServlet.service(SysServlet.java:121) at javax.servlet.http.HttpServlet.service(HttpServlet.java:805) at com.tridium.httpd.ServiceThread.handleRequest(ServiceThread.java:356)

• [web.service] Httpd request queue is full (a burst of 25 of these messages appeared at one time)

All the other messages in the log appear to be garden-variety logging, such as Saving station....

Does this help at all? I can provide a CSV of the day's logging if necessary.

Thanks, Jerry