Blog Post

#687 Haystack 4.0 Public Review

Brian Frank Thu 28 Mar

We are excited to announce the start of the Haystack 4.0 public review. Haystack 4.0 is the culmination of over a year's worth of work by WG 551.

Review Website

Haystack 4.0 includes a brand new website and documentation engine. So for the public review we are going to run the new website and documentation on a different domain - https://project-haystack.dev. We will run this website in parallel to the current website until the review process is complete. The new website has a refreshed look and feel, but will have login and forum disabled.

New Features

Haystack 4.0 includes a suite of new features to take modeling your data to the next level:

New Def Model

A new framework for the definition of tags and tag combinations is the core feature of Haystack 4.0. Defs are modeled as dicts just like all Haystack data.

New Documentation

We've built a brand new documentation engine which is completely generated from machine readable definitions. The new definition model will also feed directly into vendor tools making it easier to properly tag your data.

Symbolic Namespace

Haystack 4.0 introduces a new scalar type called Symbol used by defs to cross-reference each other.

Library Modules

All tag definitions are now scoped into modules we call libs or libraries. Libraries are versioned and easily shared as zip files. Haystack 4.0 introduces three standard libraries:

  • ph: core definitions
  • phScience: definitions related to scientific phenomenon and quantities
  • phIoT: definitions for the domain of the built environment and Internet of Things

Taxonomy

The current version of Haystack defines only a vocabulary of our terms. In Haystack 4.0 we organize the terms of our vocabulary into a tree structured taxonomy using the concept of subtyping.

Ontology

Haystack 4.0 also enables us to build more knowledge into our models by creating relationships between definitions. For example we now model containment and the flow of energy and substances in our tags. This brings Haystack into the realm of a full ontology.

Querying Enhancements

We are introducing new features into the filter query syntax to tap into the knowledge modeled by the definition framework. We can query relationships using abstract concepts without regard to exactly how the data is tagged:

  • new query terms for contains, receives, supples, conveys, and more
  • query from either endpoint of a relationship with inverses
  • transitive support to query nested relationships

RDF

We have included a proposal for how Haystack definitions and instance data is exported to RDF. This is an important new feature to ensure Haystack data is easily used with RDF tools to integrate into external ontologies.

Backward Compatibility

We have not changed how you model your data. All of your existing Haystack data models will work in Haystack 4.0. We may tweak a few of the standardized tag names, but Haystack data is still fundamentally modeled with dicts and tags just like today. Haystack 4.0 features are primarily designed to address the definition side of tag names.

Work Left

The core of the new design is complete.

A large portion of the existing tags have been redefined using the new design. However several domains are incomplete including unitary controls, lighting, VFDs, fans, pumps, etc.

One goal for Haystack 4.0 is to model spaces and zones. We have started this work by adding a new top level entity called space

Also portions of the Haystack 4.0 proposal are still in flux, specifically Generics and RDF.

Links

Important links:

Ryan Mattison Fri 29 Mar

I want to find all weather points at 90 degrees and have it return time series data for the energy consumption of each RTU etc.

Or more simple, I want to find all the temp points that have went above 90 degrees.

With the current design, I don't see a good future for using Haystack & finding trends in data.

Is this planned? Am I missing something?

Brian Frank Fri 29 Mar

Is this planned? Am I missing something?

That is exactly what people do with Haystack today. I would suggest if you have questions about how Haystack is applied that you start a new thread.

Jay Herron Mon 1 Apr

Ryan, I think maybe you missed this section

Backward Compatibility

We have not changed how you model your data. All of your existing Haystack data models will work in Haystack 4.0. We may tweak a few of the standardized tag names, but Haystack data is still fundamentally modeled with dicts and tags just like today. Haystack 4.0 features are primarily designed to address the definition side of tag names.

That means that you would tag your temp points almost exactly the same way as in the current Haystack system.

Si Chen Tue 2 Apr

This is great. I really like the sub-typing. What is the timeframe for this being official?

Coen Hoogervorst Wed 3 Apr

Hi,

Some questions. The answers might be straight forward for you guys but for me as a person just started working with Haystack some things are not really clear for me:

  • The principle of inheritance is described, which I think is a really great feature. But why is it still mandatory when I implement the tag building to a dict, to also add the tags site and space? In my opinion the advantage of inheritance is that you apply the most specific tag (as low as possible in the super/subtype tree) to a dict so that all the def's supertypes apply to the dict. For axample, I would only have to tag an air handling unit with the tag ahu which would be much easier (and not like it is described now that I also have to add the tag equip; becasue the ahu def describes "is: equip" this is allready clear.
  • Is there a list of changed tags: as I see in the documentation now, I see that hotWaterPlant is replaced by the tags hot'', water, plant and the conjuncts that result from these tags: hot-water, hot-water-plant. So some tags have changed?
  • Can you explain what the use is of the compose defs (parent/child). Is this to make a combination of tags mandatory when one of those is applied?
  • It is a little bit strange to me that in the new situation with conjuncts we have the combination of hot-water-plant and hotWaterPlantRef. In the situation of Haystack 3.0 this was the combination of the markerTag + Ref (hotWaterPlant + Ref) wich was really consistent. Still using this consistency I would expect hot-water-plantRef.

Hopefully you can explain these situations and why some of these choices have been made.

Thanx!

Stuart Longland Wed 3 Apr

The concepts of def tags sounds interesting. In PyHaystack, we're doing a kludgy form of this already… there's a stack of mix-in classes which are then "composed" into a concrete class of base type Entity based on the tags seen (i.e. if equipRef is there, EquipRefMixin gets pulled in, if his is there, it pulls in HisMixin too).

Primitive, but this does allow for a bit of flexibility. By the sounds of things, greater power will be unlocked by having a way of looking up the definitions of tags and parsing how those tags are composed.

The Relationships page covers a lot of this, and there, the example is given with the hotWaterPlantRef. So, thinking about this from the prospective of a Project Haystack client for a moment, is there going to be an operation call I can make to retrieve the def for hotWaterPlantRef? Is that what RDF addresses or is there something I missed? :-)

Joel Bender Thu 4 Apr

RDF is just a format that assumes there are "things" that you want to talk about and a really basic way to exchange data about them (and there are lots of data formats for exchanging the exact same thing, JSON-LD happens to be easy for lots of developers, Turtle is a little less syntax overhead, and RDF/XML for superlative omitted).

RDFS is a layer on top of RDF for grouping things together and has some rules for subclasses, for example, if Fido is a bush and all bushes are plants, Fido is a plant, but it doesn't say that Fido can't also be a dog.

OWL is another layer on top of that where you say that something can't be a dog and a plant at the same time. OWL is also where you can say that if something is a plant then it has roots...but not all plants have roots you say? And what about things with roots that aren't plants?

Knowing what tags are required, optional, disallowed in specific combinations, or which ones can be inferred is based on a set of conformance rules, and entailment rules can be used to "know" things that are not explicitly stated. So the concept of "hot water temperature" might be good enough for a particular context, maybe a really small Project Haystack instance.

ASHRAE 223P is in the thick of this, and a bi-directional translation from RDF to Project Haystack is a required first step in going to a Linked Data Platform.

Brian Frank Mon 8 Apr

Lots of excellent questions; answers below:

What is the timeframe for this being official?

I think that will depend on how review process goes and how quickly we can get multiple implementations as proof of concept. The way these things go in practice is that they start off with every design decision on the table and as the months go back certain things become locked down so that we can build towards stable implementations.

in PyHaystack, we're doing a kludgy form of this already

I think having the PyHaystack project become an implementation proof of concept is important. That would be good thing to discuss on another thread maybe...

The principle of inheritance is described, which I think is a really great feature. But why is it still mandatory when I implement the tag building to a dict, to also add the tags site and space?

It is true with the new def framework that the equip tag could be inferred from another tag such as ahu. However the primary goal of Haystack 4.0 was not to break how Haystack data is modeled today. We always require these tags today, so the def design includes a mechanism to ensure that supertype tags are added via mandatory. From a query and data model perspective having those tags is really convenient. I think it will take years of 4.0 rollout before we would want to consider changing anything so fundamental to how Haystack is used today.

Is there a list of changed tags

The actual defs themselves (the ontology) is still in flux. Once we have final decisions on what we want to consider changing then we will have a whole chapter dedicated those those changes. In the meantime I've been keeping notes in BitBucket

an you explain what the use is of the compose defs (parent/child). Is this to make a combination of tags mandatory when one of those is applied?

No tag is mandatory (unless its the marker tags discussed above). Tags might be required for specific applications, but from a data modeling perspective if you don't know the information then you should omit the tag. So primarily the parent/child is for docs/tools to guide users in how tags are applied to specific entity types.

It is a little bit strange to me that in the new situation with conjuncts we have the combination of hot-water-plant and hotWaterPlantRef.

Its hard to recap 18months of discussion and brainstorming on this topic. But in general the goal was to use conjuncts for the "primary dimension" of modeling. And to use camel case for the "secondary dimension". Then move the extra information about the meaning of the camel case tags into the def and make it accessible to the query engine.

Ryan Mattison Mon 22 Apr

I don't think I was very clear. The only queries on time series data I see are startDate, endDate.

Login or Signup to reply.