I would like to propose an addition to the watch subscription API; to enable records to be subscribed to by a filter.

Reason

In many implementations of user interfaces such as with dashboards and graphics, haystack filters are used to bind display objects to records such as points.

Problem

Because the watchSub op requires record id's, the applications which use filter bindings are required to resolve the filters to record ids first. This results in 2 separate network calls to subscribe to records.

If records are dynamically added to the watch, there can potentially be multiple calls for each subscription.

Current Implementation

Currently the watchSub op requires a grid with an "id" column of Ref values that looks like this:

ver:"3.0" watchDis: "my watch"
id
@id1
@id2
@id3

Proposal

I'm proposing that we also enable records to be subscribed to by a filter in addition to record ids. The grid would look like this:

ver:"3.0" watchDis: "my watch"
filter
"equipRef == @equipId and outside and air and temp and sensor"
"equipRef == @equipId and discharge and air and temp and sensor"
"equipRef == @equipId and room and air and temp and sensor"

We could even have a mixture of ids and filters like this example:

ver:"3.0" watchDis: "my watch"
filter, id
"outside and air and temp and sensor",
"equipRef == @roomEquipId and room and air and temp and sp",
,@id1
,@id2

With the subscription response, the user interface would need a way to match the binding filter it has with the result. What I've done in the past, with other ops, is that I returned the request filter with each element in the response. The response would look like this:

ver:"3.0" watchId: @myWatchId lease: 60s
id,point,curVal,outside,air,temp,sensor,requestFilter
@123,M,73,M,M,M,M,"outside and air and temp and sensor"

I think there is definitely a cool idea there

Your examples are fairly specific to matching points. So your thinking this is a filter to read one record, or that it could potentially match multiple records?

If you put a watch on a filter, would you expect the watch to automatically look for records coming into and out of the filter? For example if I put a watch on "equip" and a new equip was added, then what would the expected behavior be?

I would expect/want 2 potential solutions to up to a date filter watch list.

1. On Demand: A new watchSub with the same or updated filter would be received to the existing watch ID and that would trigger internally any equivalent watchSub/watchRemove for specific ID's to the existing list. That way the query for the filter only runs once per sub and only updates its watch list on demand.
2. Schedule: Add a new meta tag when using filter with a duration. Run the query and update the watch list at that period.

Also assume 1 can always be used even if configured for 2. Potentially reset period when on demand is used in between poll cycles although this could be server side decision not in spec.

Our case

For our current use case, each filter matches exactly 1 point. This also helps us with relativization where our filters look similar to "equipRef == $equipId and room and temp and sensor". The string interpolation happens in the client after it has queried the equipRef, then the filter that selects the point to watch looks like "equipRef == @1234 and room and temp and sensor".

Second case

I can also see a use case for filters that can select multiple points such as for an overview or point graphic where points are simply organized in a list with live values. In this case you'd watch for a specific piece of equipment with a query that looks like "equipRef == @1234 and point".

With this use case, I think that if a point that matches the filter is added after the watch is created, it would be a nice experience for that point to also dynamically show up in the UI with its live values.

Maybe in this case the watch request could look like this:

ver:"3.0" watchDis: "AHU" filter: "equipRef == @1234 and point"
empty

I think subscribing to data with a haystack filter is a great idea. The subscription response gives you all the records for that filter.

Mixing ids and filters for subscription

Should the watch subscription have a mix of filters and ids? It would be nice but I'm unsure about what the response would look like. I don't like the requestFilter column since a record could easily declare this tag itself. For instance, if a record has requestColumn, what happens in this case?

An alternative might be to declare this information in the grid's meta data. One could then say the filter outside and air matches @id1 and @id2. The grid's data itself is left alone without any naming conflicts.

Refreshing a watch

Refreshing (not polling) a watch would rerun the original filters again. After all, in a refresh you get all of the original data for the watch.

Server side filter changes

If we just keep this to the subscription and refresh, this is pretty easy to implement. If we have it so the server has to rerun the filters when it pleases (or on a refresh) then we run into some new issues. The biggest issue is the client knowing when a record is no longer being watched.

For instance, let's say a client creates a watch for outside and air. The server can rerun this filter periodically to decide what should still be watched. How does the client know about what has been unwatched?

Again we can utilize the response grid's meta data. For instance, a watch poll grid's meta data would include ids that have been added/removed since the last poll.

In regards to how the filter automatically refreshes, this could be supplied through the subscription call.

Conclusion

Modernizing the watch API with filters is a great idea. I think utilizing a response grid's meta data should be the place to include extra information about what filter maps to what id as well as what has been added/removed.

First (Major) use case

Gareth, the first use case would be to utilize filters in the same fashion that you do with ids.

In the case of graphics, imagine a gauge bound to a point with the tags: "siteRef = @1234 and elecMeter". The application sends a request to the server with about 15 filter based points. When the result comes back, it will need to match the result with the query. The "requestFilter" column provides a request <-> result relationship. The naming convention of the column could be anything, but I believe that we will need establish a request/result relationship on the first response.

The filter requests not only makes it easy to relativize the presentation, but it also makes it portable where presentations can function upon different databases without any migration.

In this use case, filters are only used during watchSub. There is no need to monitor or refresh the watch.

We can use the index to establish the request/result relationship. In this case we would need to return errors in the response for points that could not be found.

The result of a watchSub in this case would look like this:

ver: "3.0" watchId: @1234
id, point, curVal, sensor, err, errMsg
@123, M, 78, M,,
,,,,M,"Record not found"

Great idea.

One of my point is that server side should have a record limit to the query as it can be very dangerous to throw watches on 10K records within a single query.

I would want to have a client side plugin application crashing a device. A default limit should be part of the call with extensibility.

@Alper I agree for the second case.

The first case is 1 to 1. 1 filter 1 record. if it matched multiple, it will use the first.

I might try to prototype this next week, but want to formalize a few ideas...

First off, I really don't like the idea of automatically keeping track of records which enter/exit a given filtered set. It's extremely expensive to watch every single record change and then compare it against all the watched filters. And I think most of the use cases don't really require it. So I'm going to propose we nuke that idea for now and that filters are only evaluated on watchSub operation.

Second, how to unsubscribe filters? I don't want to keep track internally of filters at all beyond the initial subscribe. So I'm thinking once that initial subscription is performed, that after that it is just like you subscribed using an id. If you want to unsubscribe then you unsubscribe by id (not filter).

Third, I'd really like to take this opportunity to add a mechanism to limit which tags are sent over the wire. We've seen this can be a big a performance problem. Most of the time you really only care about changes to curVal and curStatus. I seem to recall that maybe this got hardcoded in nhaystack?

Putting it all together, here is what I propose:

Each request row in the watchSub can subscribe by id or by filter. If its by id then it works just like today. If you want to subscribe by filter then the following tags are used:

• filter: string filter
• correlateId: opaque string used in response to map back to filter used
• limit: optional number to limit recs queried

Once subscribed everything works just like today. All we are doing is enhancing the watchSub to save you a read step.

Second we define three new options you can use in the grid meta:

• allTags: marker to include all tags (default)
• onlyTags: string list of tag names
• onlyTagsPrev: marker to indicate use the last request's onlyTags list

These options can be used in the watchSub and watchPoll operations. If omitted, then the default is assumed to be allTags. With this design the server is required is keep track of the onlyTags list on a per watch basis. That allows clients to omit sending the tags list on every single poll. The id tag is always included regardless of if it is specified in the onlyTags list.

Putting it all together:

==> watchSub request
ver:"3.0" onlyTags:["curVal", "curStatus"]
filter, correlateId
"temp and sensor and point", "_sensors"
"temp and sp and point", "_setpoints"

<== watchSub response
ver:"3.0" watchId:"xyz"
id,curVal,curStatus,correlateId
@a,75.2°F,"ok","_sensors"
@b,73.1°F,"ok","_sensors"
@c,72°F,"ok","_setpoints"

==> watchPoll for only curVal,curStatus
ver:"3.0" watchId:"xyz" onlyTagsPrev
empty

Thoughts?

I think this sounds like a good enhancement that's not too difficult for folks making Haystack servers to implement. It's also backwards compatible (it doesn't break my stuff :)) which is critical.

The onlyTags feature is good idea to cut down on network traffic when polling for changes.

Eventually we do need some way to move from watches being polled to event based. Ideally a WebSocket (or HTTP server sent events) needs to be used for the watch use case. In this scenario, we won't be polling (although you still can). As such, it might make more sense to declare the onlyTags in the watchSub and for the watch to remember and use it unless otherwise changed (via another poll). Obviously the event based use case is another topic but we need to bear this in mind when making this design change to watches.