#924 Watch Subscription by filter Amendment

Michael Rochelle Fri 21 May

I would like to propose an addition to the watch subscription API; to enable records to be subscribed to by a filter.

Reason

In many implementations of user interfaces such as with dashboards and graphics, haystack filters are used to bind display objects to records such as points.

Problem

Because the watchSub op requires record id's, the applications which use filter bindings are required to resolve the filters to record ids first. This results in 2 separate network calls to subscribe to records.

If records are dynamically added to the watch, there can potentially be multiple calls for each subscription.

Current Implementation

Currently the watchSub op requires a grid with an "id" column of Ref values that looks like this:

ver:"3.0" watchDis: "my watch"
id
@id1
@id2
@id3

Proposal

I'm proposing that we also enable records to be subscribed to by a filter in addition to record ids. The grid would look like this:

ver:"3.0" watchDis: "my watch"
filter
"equipRef == @equipId and outside and air and temp and sensor"
"equipRef == @equipId and discharge and air and temp and sensor"
"equipRef == @equipId and room and air and temp and sensor"

We could even have a mixture of ids and filters like this example:

ver:"3.0" watchDis: "my watch"
filter, id
"outside and air and temp and sensor",
"equipRef == @roomEquipId and room and air and temp and sp",
,@id1
,@id2

With the subscription response, the user interface would need a way to match the binding filter it has with the result. What I've done in the past, with other ops, is that I returned the request filter with each element in the response. The response would look like this:

ver:"3.0" watchId: @myWatchId lease: 60s
id,point,curVal,outside,air,temp,sensor,requestFilter
@123,M,73,M,M,M,M,"outside and air and temp and sensor"

Brian Frank Mon 24 May

I think there is definitely a cool idea there

Your examples are fairly specific to matching points. So your thinking this is a filter to read one record, or that it could potentially match multiple records?

If you put a watch on a filter, would you expect the watch to automatically look for records coming into and out of the filter? For example if I put a watch on "equip" and a new equip was added, then what would the expected behavior be?

Jonathan Hughes Mon 24 May

I would expect/want 2 potential solutions to up to a date filter watch list.

  1. On Demand: A new watchSub with the same or updated filter would be received to the existing watch ID and that would trigger internally any equivalent watchSub/watchRemove for specific ID's to the existing list. That way the query for the filter only runs once per sub and only updates its watch list on demand.
  2. Schedule: Add a new meta tag when using filter with a duration. Run the query and update the watch list at that period.

Also assume 1 can always be used even if configured for 2. Potentially reset period when on demand is used in between poll cycles although this could be server side decision not in spec.

Michael Rochelle Mon 24 May

Our case

For our current use case, each filter matches exactly 1 point. This also helps us with relativization where our filters look similar to "equipRef == $equipId and room and temp and sensor". The string interpolation happens in the client after it has queried the equipRef, then the filter that selects the point to watch looks like "equipRef == @1234 and room and temp and sensor".

Second case

I can also see a use case for filters that can select multiple points such as for an overview or point graphic where points are simply organized in a list with live values. In this case you'd watch for a specific piece of equipment with a query that looks like "equipRef == @1234 and point".

With this use case, I think that if a point that matches the filter is added after the watch is created, it would be a nice experience for that point to also dynamically show up in the UI with its live values.

Maybe in this case the watch request could look like this:

ver:"3.0" watchDis: "AHU" filter: "equipRef == @1234 and point"
empty

Gareth David Johnson Tue 25 May

I think subscribing to data with a haystack filter is a great idea. The subscription response gives you all the records for that filter.

Mixing ids and filters for subscription

Should the watch subscription have a mix of filters and ids? It would be nice but I'm unsure about what the response would look like. I don't like the requestFilter column since a record could easily declare this tag itself. For instance, if a record has requestColumn, what happens in this case?

An alternative might be to declare this information in the grid's meta data. One could then say the filter outside and air matches @id1 and @id2. The grid's data itself is left alone without any naming conflicts.

Refreshing a watch

Refreshing (not polling) a watch would rerun the original filters again. After all, in a refresh you get all of the original data for the watch.

Server side filter changes

If we just keep this to the subscription and refresh, this is pretty easy to implement. If we have it so the server has to rerun the filters when it pleases (or on a refresh) then we run into some new issues. The biggest issue is the client knowing when a record is no longer being watched.

For instance, let's say a client creates a watch for outside and air. The server can rerun this filter periodically to decide what should still be watched. How does the client know about what has been unwatched?

Again we can utilize the response grid's meta data. For instance, a watch poll grid's meta data would include ids that have been added/removed since the last poll.

In regards to how the filter automatically refreshes, this could be supplied through the subscription call.

Conclusion

Modernizing the watch API with filters is a great idea. I think utilizing a response grid's meta data should be the place to include extra information about what filter maps to what id as well as what has been added/removed.

Michael Rochelle Tue 25 May

First (Major) use case

Gareth, the first use case would be to utilize filters in the same fashion that you do with ids.

In the case of graphics, imagine a gauge bound to a point with the tags: "siteRef = @1234 and elecMeter". The application sends a request to the server with about 15 filter based points. When the result comes back, it will need to match the result with the query. The "requestFilter" column provides a request <-> result relationship. The naming convention of the column could be anything, but I believe that we will need establish a request/result relationship on the first response.

The filter requests not only makes it easy to relativize the presentation, but it also makes it portable where presentations can function upon different databases without any migration.

In this use case, filters are only used during watchSub. There is no need to monitor or refresh the watch.

Michael Rochelle Tue 25 May

We can use the index to establish the request/result relationship. In this case we would need to return errors in the response for points that could not be found.

The result of a watchSub in this case would look like this:

ver: "3.0" watchId: @1234
id, point, curVal, sensor, err, errMsg
@123, M, 78, M,,
,,,,M,"Record not found"

Alper Üzmezler Fri 28 May

Great idea.

One of my point is that server side should have a record limit to the query as it can be very dangerous to throw watches on 10K records within a single query.

I would want to have a client side plugin application crashing a device. A default limit should be part of the call with extensibility.

Michael Rochelle Thu 3 Jun

@Alper I agree for the second case.

The first case is 1 to 1. 1 filter 1 record. if it matched multiple, it will use the first.

Login or Signup to reply.