We recently upgraded Tridium from v3.5 to v3.8. nHaystack authentication from SkySpark to Tridium was working fine on the old 3.5 server, but now the same account is receiving a "Basic authentication failed [302]" error. However, when we login to Haystack through a web browser with the same credentials, the nHaystack details come up successfully.

All the web service and user settings are the same on both the 3.5 and 3.8 stations. On 3.5, the authentication scheme was set to "Cookie". On 3.8, after I changed the scheme to "Basic", the SkySpark Haystack connector's authentication error changed to "Basic authentication error [401]", and in the web browser, login attempts with the same credentials failed with a "401 - Access Denied" error.

Ultimately, is there any additional authentication settings I need to apply/change in Tridium 3.8 for the nHaystack 1.2.4 service to work? Would upgrading to the latest version of nHaystack resolve this issue?

Thanks, Jerry

Hi Jerry,

I've done a quick, and reasonably un-scientific test, by running a station under 3.8 and trying both Cookie and Basic Authentication and logging in through the web browser.

I did this with both nhaystack 1.2.4 and 1.2.5. Is there any other information you can provide? Is there an error in the Application Director of the station?

regards, Richard

Richard, Thanks for your testing. I have no problems either when logging into nHaystack on Tridium through a web browser. The authentication problem only resides in SkySpark, when I try to use SkySpark's Haystack connector. Within SkySpark's Connectors app is where I see the basic authentication error, which I can reproduce by simply pinging the Haystack connector.

I did not see any error message on the Tridium platform, but there may be some more details on the SkySpark console, which I did not have access to at the time. I'll dig up the SkySpark logs and see if there's anything more specific in there.

Jerry

OK, I've reviewed the SkySpark logs for the customer project but they don't have any record of my authentication attempts or failures. I couldn't review the SkySpark Host-level logs, but I wouldn't expect them to have any nHaystack records either, as the Haystack connector is loaded to the project.

Is there a way to increase the logging level of the nHaystack driver on the Tridium side? Is there any difference between logging in to nHaystack through a web browser and through SkySpark's Haystack connector?

Thanks, Jerry

There are some logs for nhaystack which you can configure from the Spy\Log Setup page in your station.

If you set them all to "trace" level you will get all log messages that nhaystack puts out.

I also ran up the latest SkySpark on my own laptop and connected it to a Niagara station running nhaystack 1.2.4 and 1.2.5 and using both Cookie and Basic Authentication I was able to get a connection "ping"-ing. I didn't do anything further than that though.

Specifically which version of 3.8 are you using. I'm wondering if there are some differences in authentication between say 3.8.38 and 3.8.41. For the record I'm running 3.8.41 with all the latest patches from Tridium.

@Brian: is there any other debugging we can put on the SkySpark side?

Richard

The easiest way to debug the full HTTP requests/responses for authentication from SkySpark is to use the command line program:

fan haystack::Client http://localhost:8080/api/demo username password

Thanks for both of your help. Using the Fantom call, I was able to get a fuller description of the error. It looks like the authentication error is "302: Moved Temporarily", which makes sense because Tridium itself just migrated to a new VM.

I'm trying to confirm if the nHaystack.jar file was simply copy-and-pasted from the old Tridium VM to the new one, and if so, my hunch is the nHaystack driver has retained some of the settings from its old home. If this is the case, I plan to do a fresh install of nHaystack v1.2.4 on the new VM, and if that doesn't work, try installing v1.2.5.

Other than the .jar file, is there anything else I need to remove in order to completely remove any traces of nHaystack from Tridium? Specifically, is there a way to clear the nHaystack cache, as well as any other settings that Tridium may keep on nHaystack? I want to ensure any traces of nHaystack are out before I re-install the fresh version of it.

Lastly, the customer is now using Tridium v3.8.38. They had upgraded from Tridium v3.5.

Thanks, Jerry

Update: No luck using a fresh install nHaystack 1.2.4 or 1.2.5, as SkySpark continues to receive the same 302 error. Looking into the 302 error further, it looks more like a redirect issue than an authentication issue.

I would suggest posting a full dump of the HTTP requests and responses. Authentication with Niagara is extremely fragile because there isn't any information provided in their responses about how to authenticate.

OK, here's the full output of the Fantom haystack authentication call:

[14:06:11 29-Feb-16] [debug] [haystackClient] > [0]
GET http://AX-SERVER/haystackNew/about
Accept-Encoding: gzip

[14:06:11 29-Feb-16] [debug] [haystackClient] < [0]
302 Moved Temporarily
cache-control: private, must-revalidate
content-type: text/html; charset=UTF-8
expires: Thu, 01 Jan 1970 00:00:00 GMT
set-cookie: niagara_session=s90e6e1676d3d75cc3227efc5f8656518a1f7b489c07e111d5e; path=/,niagara_uri=/haystackNew/about; path=/
location: http://AX-SERVER/login
content-length: 889
server: Niagara Web Server/3.8.38
date: Mon, 29 Feb 2016 22:06:10 GMT
<html>
<head>
<meta name='viewport' content='width=device-width initial-scale=1.0 maximum-scale=1.0 target-densityDpi=medium-dpi' />
</head>
<body>
<h1>302: Moved Temporarily</h1>
<p>The above error occurred while processing your request.</p>
<button style='font-size: 150%; font-weight: bold;' onclick='history.back();'>Back</button>


</body>
</html>

[14:06:11 29-Feb-16] [debug] [haystackClient] > [1]
GET http://AX-SERVER/haystackNew/about
Cookie: niagara_uri=/haystackNew/about; niagara_session=s90e6e1676d3d75cc3227efc5f8656518a1f7b489c07e111d5e
Authorization: Basic c2t5c3Bhcms6TiFAZ2FyYTIwMTQhIQ==
Accept-Encoding: gzip

[14:06:11 29-Feb-16] [debug] [haystackClient] < [1]
302 Moved Temporarily
cache-control: private, must-revalidate
content-type: text/html; charset=UTF-8
expires: Thu, 01 Jan 1970 00:00:00 GMT
set-cookie: niagara_uri=/haystackNew/about; path=/
location: http://AX-SERVER/login
content-length: 889
server: Niagara Web Server/3.8.38
date: Mon, 29 Feb 2016 22:06:10 GMT
<html>
<head>
<meta name='viewport' content='width=device-width initial-scale=1.0 maximum-scale=1.0 target-densityDpi=medium-dpi' />
</head>
<body>
<h1>302: Moved Temporarily</h1>
<p>The above error occurred while processing your request.</p>
<button style='font-size: 150%; font-weight: bold;' onclick='history.back();'>Back</button>


</body>
</html>

haystack::AuthErr: Basic authentication failed [302]
  haystack::BasicAuthAlgorithm.auth (ClientAuth.fan:232)
  haystack::ClientAuth.auth (ClientAuth.fan:45)
  haystack::Client.openx (Client.fan:39)
  haystack::Client.main (Client.fan:299)
  java.lang.reflect.Method.invoke (Unknown)
  fan.sys.Method.invoke (Method.java:559)
  fan.sys.Method$MethodFunc.callList (Method.java:198)
  fan.sys.Method.callList (Method.java:138)
  fanx.tools.Fan.callMain (Fan.java:183)
  fanx.tools.Fan.executeType (Fan.java:147)
  fanx.tools.Fan.execute (Fan.java:41)
  fanx.tools.Fan.run (Fan.java:308)
  fanx.tools.Fan.main (Fan.java:346)

It basically looks like they are ignoring the Basic Authentication and just sending another redirect to the login page. I would double check that Basic authentication is still enabled and perhaps ensure it is still supported in 3.8. If you think it is still enabled correctly, then you might have to work with Tridium to check if they did something to disable support for it.

I know this may sound obvious... but did you check that the servlet was activated ? I did see this error and the servlet was deactivated...

Re-enabled, rebuild cache and everything was ok again...

During our troubleshooting, we have:

• Disabled and re-enabled the nHaystack servlet
• Changed the servlet name
• Removed and added the trailing backslash in the SkySpark Connector's URI string
• Changed the Tridium web service port number
• Disabled and re-enabled the Http Enabled setting in Tridium's web service
• Disabled the Https Enabled and Https Only settings in Tridium's web service
• Tried all three Authentication Scheme's (Cookie, Cookie Digest and Basic) in Tridium web service. The authentication that worked with nHaystack on the old Tridium v3.5 server was 'Cookie', and it now gives us the [302] errors. Changing the authentication to 'Basic' gives us [401] errors instead.
• Installed fresh installations of nHaystack 1.2.4 and 1.2.5
• Rebuilt the nHaystack cache several times

Throughout nearly every iteration of the steps above, logging into nHaystack using a web browser still works (changing the authentication scheme screws up the browser login), while the SkySpark connector still fails every time with the 302/401 error.

For comparison purposes, here's the stack trace of a successful authentication call to our own nHaystack service, running on Tridium v3.5 and uses Cookie authentication in its web service:

[14:52:34 29-Feb-16] [debug] [haystackClient] > [0]
GET http://tridium-energy/haystack/about
Accept-Encoding: gzip

[14:52:34 29-Feb-16] [debug] [haystackClient] < [0]
302 Moved Temporarily
date: Mon, 29 Feb 2016 22:52:24 GMT
content-length: 120
set-cookie: niagara_uri=/haystack/about; path=/
server: Niagara Web Server/3.7.106.3
expires: Thu, 01 Jan 1970 00:00:00 GMT
content-type: text/html; charset=UTF-8
location: http://tridium-energy/login
cache-control: private, must-revalidate
<html>
<body>
<h1>302: Moved Temporarily</h1>
The above error occurred while processing your request.
</body></html>

[14:52:34 29-Feb-16] [debug] [haystackClient] > [1]
GET http://tridium-energy/haystack/about
Authorization: Basic YWRtaW46QWxlcnRvbjE5ODY=
Cookie: niagara_uri=/haystack/about
Accept-Encoding: gzip

[14:52:34 29-Feb-16] [debug] [haystackClient] < [1]
200 OK
date: Mon, 29 Feb 2016 22:52:24 GMT
set-cookie: niagara_session=sfc73d48db8cb5ce7e370efe19233e38ac8338bb545ce9a2fae; path=/
server: Niagara Web Server/3.7.106.3
expires: Thu, 01 Jan 1970 00:00:00 GMT
content-type: text/plain; charset=utf-8
connection: close
cache-control: private, must-revalidate
ver:"2.0"
haystackVersion,serverTime,moduleName,serverBootTime,serverName,productVersion,moduleVersion,productUri,tz,moduleUri,productName
"2.0",2016-02-29T14:52:24.702-08:00 Los_Angeles,"nhaystack",2016-02-10T03:31:11.794-08:00 Los_Angeles,"ATS_LOG_HOST","3.7.106","1.2.4",`http://www.tridium.com/`,"Los_Angeles",`https://bitbucket.org/jasondbriggs/nhaystack`,"Niagara AX"

[14:52:35 29-Feb-16] [debug] [haystackClient] > [2]
POST http://tridium-energy/haystack/about
Authorization: Basic YWRtaW46QWxlcnRvbjE5ODY=
Content-Length: 17
Content-Type: text/zinc; charset=utf-8
Accept-Encoding: gzip
ver:"2.0"
empty

[14:52:35 29-Feb-16] [debug] [haystackClient] < [2]
200 OK
date: Mon, 29 Feb 2016 22:52:24 GMT
set-cookie: niagara_session=s9d9e840c043be8f64ae16409b57cd281b17f07a15381deaa7f; path=/
server: Niagara Web Server/3.7.106.3
expires: Thu, 01 Jan 1970 00:00:00 GMT
content-type: text/plain; charset=utf-8
connection: close
cache-control: private, must-revalidate
ver:"2.0"
haystackVersion,serverTime,moduleName,serverBootTime,serverName,productVersion,moduleVersion,productUri,tz,moduleUri,productName
"2.0",2016-02-29T14:52:24.796-08:00 Los_Angeles,"nhaystack",2016-02-10T03:31:11.794-08:00 Los_Angeles,"ATS_LOG_HOST","3.7.106","1.2.4",`http://www.tridium.com/`,"Los_Angeles",`https://bitbucket.org/jasondbriggs/nhaystack`,"Niagara AX"


Ping successful: http://tridium-energy/haystack/

haystackVersion:  2.0
moduleName:       nhaystack
moduleUri:        https://bitbucket.org/jasondbriggs/nhaystack
moduleVersion:    1.2.4
productName:      Niagara AX
productUri:       http://www.tridium.com/
productVersion:   3.7.106
serverBootTime:   2016-02-10T03:31:11.794-08:00 Los_Angeles
serverName:       ATS_LOG_HOST
serverTime:       2016-02-29T14:52:24.796-08:00 Los_Angeles
tz:               Los_Angeles

The successful and failed calls look like they start to differentiate in step [1]. The same calls are made, and locally nHaystack returns "200 OK" while at our customer's site, it returns "302 Temporarily Moved".

Does this help with diagnosing the problem at all? I'm running out of ideas to tackle this myself.

Jerry

Does this help with diagnosing the problem at all? I'm running out of ideas to tackle this myself.

It just looks like its not authenticating with Basic anymore - which is essentially the only way to authenticate with AX. Really your only recourse is to work with Tridium and find out if a) they disabled basic auth and b) what they expect you to use

OK, good news. I installed a fresh version of nHaystack on a Tridium AX v3.8.38 on a tester laptop, and was able to successfully connect SkySpark to nHaystack on the Tridium server.

I've successfully tested both nHaystack v1.2.4 and 1.2.5, and have connected SkySpark to Tridium/Haystack both locally (on the same computer) and remotely (SkySpark on a different laptop, connecting via local network). I used the boilerplate Tridium settings, nothing fancy.

Thus, I am fairly certain that this isn't a compatibility bug between nHaystack and Tridium, nor a Basic Authentication bug, but probably something in the customer's network/domain security. I'll follow up if/when we find the culprit/setting.

Jerry

OK, I'm fairly certain I've found the bug that is preventing SkySpark / Haystack communication with the Tridium AX v3.8. I've tested various LDAP configurations of Tridium's user service all day and found that the latest version of the LDAP User Manager does not allow for SkySpark connectivity to the nHaystack web service, failing at the authentication step no matter what type of user credentials are entered, be it a LDAP user or local Tridium user.

I've found a hacky workaround to get nHaystack / SkySpark communication working again. I replaced the ldap.jar file in the Niagara-3.8.41/modules folder with the same file from the Niagara-3.5/modules folder, reverting the LDAP component back to its v3.5 state.

The good news with this hack is nHaystack / SkySpark communication is re-established, as long as SkySpark uses a Tridium local user account, not an account stored in LDAP. This is how SkySpark is already set up.

The bad news with this hack is authentication of any LDAP users fails. Any LDAP user login attempts fail with this message logged to Application Director: "ERROR [sys.service] ldap:LdapUserService does not have an associated AuthAgent.". My guess is this AuthAgent is a feature added after v3.5, and the UserManager depends on it.

I've tried using both nHaystack v1.2.4 and 1.2.5, and both have the same problem. And the LDAP "hooks" to an LDAP server do not need to be configured for this communication problem to occur. Enabling the LDAP user service module and using only local user accounts still prevent SkySpark / nHaystack communication.

Our customer may be willing to limit their authentication to local users temporarily. Any chance someone can dig into this issue and let me know if this is a nHaystack or Tridium-side issue?

Thanks, Jerry

Any chance someone can dig into this issue and let me know if this is a nHaystack or Tridium-side issue?

Authentication doesn't have anything to do with nHaystack, its handled by Niagara before the nHaystack servlet is even accessed. So you'd have to work with Tridium and figure out why Basic Auth isn't working with LDAP

OK, thanks for the confirmation Brian. Asked in a slightly different way, is there any way nHaystack can handle this bug, or is this strictly a Tridium-side fix?

Jerry

When trying to connect Niagara AX 3.8 by using haystack connector, error 410 and 302 is occurring.

Here is the details.

id: 1eab2c19-eceb9bb7 dis: Haystack Conn status: fault connPollFreq: null connLinger: 30sec lingering: -2138426599317ns openPins [,] actorTimeout: 1min lastPing: never lastConnOk: never lastConnFail: 36min 8sec ago lastPoll: never pollFreq: 5sec points: 0 pointsInWatch: 0 curErr:

haystack::AuthErr: Basic authentication failed [410]

haystack::BasicAuthAlgorithm.auth (ClientAuth.fan:383)
haystack::ClientAuth.authHaystack (ClientAuth.fan:79)
haystack::Client.openx (Client.fan:39)
haystackExt::HaystackConn.onOpen (HaystackConn.fan:60)
connExt::Conn.open (Conn.fan:150)
connExt::Conn.openLinger (Conn.fan:116)
connExt::Conn.openLinger (Conn.fan)
connExt::Conn.ping (Conn.fan:204)
connExt::Conn.receive (Conn.fan:318)
haystackExt::HaystackConn.receive (HaystackConn.fan:44)
connExt::ConnActor.receive (ConnActor.fan:131)
concurrent::Actor._dispatch (Actor.java:230)
concurrent::Actor._work (Actor.java:201)
concurrent::ThreadPool$Worker.run (ThreadPool.java:262)

Is there any help to establish successful connection?.

Thanking you,

Kathiresan Rajagopal

Kathiresan, WIth Tridium AX 3.8, only local user authentication with work with nHaystack. Or, in other words, you can't use LDAP / Active Directory users on the Triduim AX. In fact, I think if you even enable the ldapAuth module on the Tridium AX, basic authentication with a local Tridium user will fail. So, first things I'd check arewhat type of user account you are using to connect SkySpark to nHaystack, and what type(s) of authentication methods are being used or are installed on the Tridium AX.

Jerry

Brian,

This used to work in 2.1.15:

fan haystack::Client http://localhost:8080/api/demo username password

But in 3.0.12, we get an error:

ERROR: sys::UnknownTypeErr: haystack::Client

Is there a newer version of "haystack::Client"?

In 3.0 it is:

fan skyarcd::Client <uri> <user> <pass>