Sub Meter Data Import & Storage Platform RFP Questions/Answers

Transcription

1 Sub Meter Data Import & Storage Platform RFP Questions/Answers ADDED 10/12/2015 Q: The latter sections of the RFP indicate that you are looking for dashboarding features. Will VEIC accept a proposal that excludes that part? A: The primary functionality that VEIC seeks is a sub metering data acquisition and storage system. VEIC will evaluate solutions that do not contain a dashboard feature and take that into consideration when comparing proposals. 10/7/2015 Page #3 Scope of Work Q: The Scope of Work identifies that VEIC will be hosting the system, and the assumption is providing the required server resources. What level of access will be available to available to the vendor for troubleshooting, diagnostics and maintenance such as server patches i.e. access to log files? Will VEIC be responsible for system maintenance, software upgrades etc.? A: VEIC has no preference towards a vendor hosted or in house hosted solution. If the proposal calls for a VEIC hosted solution, then the vendor will have complete access to the servers during development. VEIC will maintain the system post implementation and will work jointly on future upgrades. Q: Page # Describe the process for adding new data fields, explaining whether that process can be completed by VEIC staff or if it requires custom development. Can VEIC provide clarification on the type of data fields that would need to be added? A: VEIC is interested in tracking data about projects, the metering equipment installed, and the physical location and building systems on which it is installed. Data fields to be added might include text or numeric fields that can be used to store structured data like project type (residential/commercial), meter configuration parameters (e.g. CT size and scaling), and installation details like room number or building equipment type. Q: Page # Describe mechanisms for ensuring data completeness and integrity, including backups. Will VEIC be responsible for maintaining backups and a redundant server if needed? A: If VEIC hosts the solution, VEIC will back up the system and provide a failover server as well, if appropriate. Please describe any mechanisms your solution can provide that can create backups of the data, not just the whole server. VEIC can run those systems, but would be interested to know if there are dedicated systems that manage data integrity above and beyond whatever backup mechanisms would be provided by the DBMS. Q: Page # VEIC Sub Metering Project and Meter Management. Can VEIC please provide details on the existing Sub Metering Project and Meter Management system. i.e. commercial product or in house developed system. Are there existing or preferred API interfaces to this system?

2 A: VEIC has an in house project management and savings tracking system built on a web platform. The current inventory system utilizes an Excel spreadsheet to manually track VEIC s meter supply and project status. There are no existing APIs to interface with the system. Q: Page # VEIC Analytics. Can VEIC, please provide details on the existing VEIC Analytics applications i.e. commercial product(s) or in house developed applications? Will vendor need to support all applications or will there be a common API established and VEIC would be responsible supporting exchanges with various applications? A: VEIC will work with the successful bidder to jointly develop an API that will meet all of VEIC s data exchange needs. The API will require reading and writing of data between systems. Q: Is VEIC open to a vendor custom building a platform or do you expect one off the shelf? A: VEIC will consider custom built solution, off the shelf solution, or a combination of the two. Q: In Task 1, item 1.3 requests: "Describe the process for adding new data fields, explaining whether that process can be completed by VEIC staff or if it requires custom development." Please provide representative examples of the types of new data fields VEIC expects it may need to add. A: VEIC is interested in tracking data about projects, the metering equipment installed, and the physical location and building systems on which it is installed. Data fields to be added might include text or numeric fields that can be used to store structured data like project type (residential/commercial), meter configuration parameters, and installation details like room number or building equipment type. 0.1 FUNCTIONALITY 0.2 Describe Data movement requirements Describe the requirements for moving data into and out of the Sub Metering database system (SMDB), such as: batch or real time import/export, replication, database copy etc. A: The ideal solution should support batch and real time import/export. Section 2.1 requests the respondent to describe what their solution can provide. 0.3 What Data formats will be supported? Describe the data formats required to be supported by the solution, and what that support entails. Please explicitly describe requirements for: Structured data Text Image Binary Key value Graphical XML RDF JSON Avro Other (specify)

3 A: We have established requirements for CSV, XML, JSON, as well as a desire to incorporate images. Other formats may be desirable in the future. 0.3 B Is there historical data to be loaded? If so please describe the format, volume, and number of formats for the historical data A: VEIC currently stores historical data in CSV format and some Excel. 0.3 C For each data format, describe the requirements for speed of data ingestion. Real Time Intraday Batch Daily Batch Other A: For data that is being pulled from internet connected devices, VEIC expects to have current data that is no older than 48 hours. For data that is being uploaded through a user interface, VEIC requires a user interface that provides a standard experience of latency and responsiveness. 0.4 Describe the requirements to retrieve data, such as: SQL (specify variant and version, and key supported features including level of support for ISO/IEC 9075 and support for HDFS) XQuery SPARQL Full text search Image search Other (specify) 0.5 Data registration requirements within (SMDB) Describe any requirements to register datasets as or after they are loaded into the SMDB. A: VEIC has no requirements at this time. Can discuss further during scoping with the successful bidder. 0.6 Data registration external Describe any requirements to register external data so it can be used in conjunction with data within the SMDB. A: VEIC has no requirements at this time. Can discuss further during scoping with the successful bidder. 0.7 Schema Management. Even though data may be loaded into the SMDB without specifying their format in advance, in order to process the data, their format needs to be known. If the data is structured in any way, it needs a corresponding schema definition. Describe any requirements for managing schema definitions within the SMDB. 0.8 Schema Management 3rd party integration Describe requirements for managing any 3rd party schema definitions within the SMDB. 0.9 Schema design

4 Describe any schema designs requirements, such as: Relational: 3NF, star schema,.snowflake schema, data vault; XML schema, RDF schema, graph schema Other (specify) Industry data models Describe any requirements for 3rd party Utilities and/or Energy industry data models that can be used with the product. Describe current data model being used. A: VEIC has no firm requirements but is investigating the use of Haystack, haystack.org/ 0.11 Data Summarization Describe requirements for aggregating and/or summarizing data. A: Data needs to be aggregated and summarized by time (daily, hourly, etc.), both by summing and averaging values. It may be desirable to aggregate channels or provide other statistical metrics when summarizing. 0.12a Data Analytics Describe requirements for data analytics. A: VEIC maintains its own data analytics systems. Section 3 of the RFP requests the respondent to describe the analytic and data viewing capabilities native to the system. 0.12b Data Analytics Describe SMDB integration requirements with VEIC s Project Mgt. and Savings Tracking system. A: System needs to track one or more foreign keys from VEIC s system and use those keys when selecting data through the APIs Data Governance Describe requirements for data governance, including data stewardship and data provenance. A: VEIC expects a secure solution that protects the privacy of the data collected and stored. Please see 1.2 and describe how your solution will accomplish that Data Visualization Describe requirements including but not limited to: Basic Query and reporting OLAP Data Mining Dashboard Report (s) Data Sandbox requirements BI Tools requirements Other A: Please see 3.2 for examples of data visualization. Please propose a solution that fits those examples Data Visualization Describe data consumer profile: Power User

5 Analyst Developer End user A: VEIC Engineering, Commercial Account Management, and Measurement & Evaluation staff will be the consumer of the data visualization. All have deep knowledge of energy consumption patterns Data Visualization Describe user tasks: View Analysis Patterns Report Aggregated Interactive Other Specify? A: Please see for questions on solutions capabilities Other functionality Describe any other significant functionality not covered above. A: Additional functionality can be discussed with the successful proposer during scoping and contract negotiations TECHNCAL ARCHITECTURE Overview Describe current technical architecture in place. Does bidder have to adhere to already existent architecture best practices? A: VEIC would provide the hardware and virtual machines that are needed to run the solution. VEIC would work with vendor to specify the system requirements. VEIC typically operates in a hardware environment that is Microsoft based, but VEIC would consider a LINUX solution as well Administration Initial Configuration Describe requirements on initial configuration of the solution, with details on different types of applications API s/projects/use case(s) that will be supported. A: To be discussed with the successful proposer during scoping and contract negotiations Administration Ongoing maintenance Describe requirements for ongoing maintenance of the solution, with examples of the staffing expectations, typically required for different types of applications API s// projects / use cases. A: VEIC does expect ongoing support. VEIC recognizes that the support would vary depending on whether the successful bid is a VEIC or vendor hosted solution. Please describe your standard levels of support for clients Administration Monitoring tools Describe tool requirements to monitor the solution, and how they may be typically used. A: VEIC will be responsible for monitoring performance of the solution Administration Backup / Restore

6 Describe the requirements to backup data and restore it, with examples of when such facilities would typically be used. (E.g. to be able to reverse bad updates; to be able to create a new test environment) A: If VEIC hosts the solution, VEIC will back up the system and provide a failover server as well, if appropriate Administration High availability Describe requirements for high availability. Describe maintenance and any circumstances under which outages may be required. A: The acquisition of sub metering data should have provisions for collecting data in batch mode when the system cannot reach the sub meters. An uptime of 99% will be adequate if the system is able to fill in data gaps once it comes back online Administration Disaster Recovery Describe requirements for disaster recovery capabilities with focus on options available for hot, warm and/or cold failover Administration Ease of upgrades Describe the requirements for typical frequency of patches or other upgrades, and what is involved in applying them. What approach is typically taken to testing upgrades before applying them? A: Frequency of patches will be dictated by the security needs for the patch. VEIC utilizes a test and production environment in order to test and apply patches and upgrades. Feature and functionality upgrades to the solution will be directed by the needs of the sub metering business program as well as any continuous system improvements offered by the vendor Development and Testing Describe options for providing development and testing environments with focus on Adaptors and Importers that will import data from the meters and ingest into SMDB. How are these Adaptors and Importers slated to be used? A: The solution should already provide mechanisms to acquire data from popular sub meter devices. See 2.2 for examples Development and Testing Describe API development requirement(s) in relation data ingestion, curation and presentation. Describe system integration requirements for SMDB A: The solution should already provide mechanisms to acquire data from popular sub meter devices. See 2.2 for examples Metadata management Describe any requirements on managing metadata. A: Please see 2.4 for examples of metadata and describe how your solution will collect and store that data Metadata management Other integration Describe any requirements for integrating with any other metadata management tools. A: Please see 2.4 for examples of metadata and describe how your solution will collect and store that data. Currently, other metadata management tools are custom developed by VEIC, so integration

7 requirements can be implemented as part of the API and requirements developed with the successful bidder Performance Batch loads Describe any performance requirements for batch data loads. A: Please see 2.3 for examples of devices that require batch data loads and describe how your solution will collect and store that data. For data that is being pulled from internet connected devices, VEIC expects to have current data that is no older than 48 hours. For data that is being uploaded through a user interface, VEIC requires a user interface that provides a standard experience of latency and responsiveness Performance Data updates Describe any requirements for optimizing data updates. What throughputs are typical for various configurations? A: Currently, VEIC is sizing the system for less than 1,000 active projects, each with <100 meter channels (including per phase power, power factor, etc.), typically collecting data at 1 5 min intervals. Data may be batch uploaded nightly for internet connected systems. For offline systems, a collection period of 2 8 weeks would typically be uploaded upon retrieving the meters Performance Data queries Describe any requirements for optimizing data queries. What throughputs are typically required for various configurations? A: Analysis queries typically use 5 20 channels and could require 1 7 days of 1 min interval data or up to a year of 15 min interval data. Downloading could take place over several minutes. Please explain cost/performance trade offs if known Performance Data Streaming Describe any requirements to stream data loads. A: Please propose a solution; see details in Question data updates answer Performance in memory processing Describe the requirements to process data in memory and when they would typically be used Scalability Describe the requirements for both horizontal and vertical scalability. (Provide your definitions for the terms horizontal and vertical.) Scalability Data storage capacity Describe the range of storage capacities needed. A: Please propose a solution; see details in Question data updates answer Scalability Compute capacity Describe the requirements for compute capacities needed.

8 Scalability Compute capacity expansion Describe the requirements for ease with which compute capacity can be expanded, Do we need to adhere to any predefined process(s) for doing so and the supported increments? Platforms on premise solution Describe the hardware and operating systems which are available for an on premise solution, and the criteria typically used for choosing among them. Do we need commodity and/or specialized hardware? A: VEIC would provide the hardware and virtual machines that are needed to run the solution. VEIC would work with vendor to specify the system requirements. VEIC typically operates in a hardware environment that is Microsoft based, but VEIC would consider a LINUX solution as well Platforms Cloud solution Describe the requirements for a cloud solution, and the criteria typically used for choosing among them Platforms that work with the solution Describe other requirements for platforms that the solution need to work with, e.g. RDBMS, Data Analytics, and BI reporting A: The primary interface with the system will be through web interfaces and APIs. Direct access through RDBMS queries are desirable but not necessary Platforms cloud integration Describe requirements to integrate with other data in the cloud, including: MS Azure, IBM Soft layer, Salesforce, WorkDay, other (specify). A: There are no requirements for the system to integrate with other, non VEIC systems Security Describe the requirements for: authentication, authorization, segregation and auditing/logging. A: VEIC expects a secure solution that protects the privacy of the data collected and stored. Please see 1.2 and describe how your solution will accomplish that Encryption Describe the requirements for encrypting data in motion and at rest. A: VEIC expects a secure solution that protects the privacy of the data collected and stored. Please see 1.2 and describe how your solution will accomplish that Interoperability Other Describe any other interoperability of significance. A: N/A Technical Architecture Other Describe any other significant features of the architecture not already covered above. A: N/A IMPLEMENTATION AND SUPPORT Implementation process

9 Describe your requirements for typical implementation process (s) on small, medium and large implementations, stating any assumptions. A: VEIC will work with the successful bidder on a mutually acceptable project plan Implementation staffing Describe the types and levels of staffing typically available during the implementation A: VEIC will work with the successful bidder on a mutually acceptable project plan Implementation support Describe the requirements on types and levels of support to assist with the implementation of the solution, stating any assumptions. Document the availability of on site and remote support by geography, especially for: remote offices if any A: VEIC is headquartered in Burlington VT. All staff working on the project will be located here Implementation duration Describe the requirements for typical duration of the implementation for small, medium and large installations, stating any assumptions. A: VEIC will work with the successful bidder on a mutually acceptable project plan Ongoing support Describe requirements for levels of support needed for ongoing support of the solution, stating any assumptions. A: VEIC does expect ongoing support. VEIC recognizes that the support would vary depending on whether the successful bid is a VEIC or vendor hosted solution. Please describe your standard levels of support for clients Training virtual Describe the requirements of virtual classroom and self paced training, and the languages in which it is required. Provide a list of courses. A: There are no requirements at this time and is up to the bidder to propose Training classroom Describe the requirements of physical classroom training, and the languages required to be supported A: English is the only language required. There are no other requirements at this time and is up to the bidder to propose 2) In Task 5.1, will the Contractor be responsible for configuring submetering equipment for deployment? A: The Contractor need not be directly responsible for configuring equipment, but the proposed solution should have mechanism to configure equipment. If equipment configuration services can be provided as part of the solution, please describe and price separately. VEIC staff should be able to configure equipment as well. See 5.1 3) In Task 5.1, will the Contractor be responsible for deployment of submetering equipment in the field? A: No. VEIC will manage the equipment. 4) In Task 5.2, will the Contractor be responsible for storing and managing sub metering equipment not deployed?

10 A: No. VEIC will manage the equipment. 5) In Task 5.3, will the Contractor be responsible for planning for each customer site with the various interested parties what will be measured, which sub metering equipment to be utilized, and the optimal configuration for each measurement? A: No. VEIC will manage the sub metering programs and decide which equipment to use. 6) Is it anticipated that VEIC will want to import metering data from utility billing meters as well as submetering data? Or will VEIC be working only with submetering data? A: Utility bill meter data could be one of the additional data sources that the solution imports. It would be provided via API or CVS/Green Button batch upload. See task ) Does Task 2 only include 2.1 through 2.5? The Power Point provided with the RFP references tasks 2.3 to 2.8. A: Task 2 only includes 2.1 through 2.5. The diagram reflects an older list and should say, (see task ) 8) General Who is/are the existing vendor(s) for meter management and other metering programs? A: VEIC currently uses a series of manual process to manage its meter programs. 9) General What is the budget for the Sub metering Data Platform during the first year? A: The budget has not been determined yet. 10) Project Overview Please describe the desired functionality that is missing from your existing meter project and inventory system? A: The current inventory system utilizes an Excel spreadsheet to manually track VEIC s meter supply and project status. 11) Project Overview What is the existing software that comprises VEIC Analytics Applications? A: VEIC has built an in house analytics solution and manipulates its data with various analytics tools including Python, R, and Mathematica. 12) Project Overview What is the existing software that comprises VEIC Project Management and Savings Tracking System? A: VEIC has an in house project management and savings tracking system built on a web platform. 13) Data Import Functionality How will the geolocation data in be utilized? A: Geolocation data will be utilized by the Sub Meter Team to document the physical location. The ability to show projects on a map is desirable, but not required. 14) Data Import Functionality How will the photos (image files) in be utilized? A: Photos will be utilized by the Sub Meter Team to document the physical location. The ability to view photos when browsing meter projects is desirable, but the ability to download photos through the web interface and/or API is required. 15) VEIC Analytics Is this functionality part of the new system or an enhancement to the existing system? A: VEIC has its own Analytics Tools.

11 16) VEIC Analytics Please provide examples of the questions that the analytics will be required to answer? A: VEIC has its own Analytics Tools. 17) Pricing Of the 150 web connected meters, how many different protocols/file types/file formats will be employed? A: The majority of the web connected meters will be egauage. A smaller number will be Dent Elite Pro SP and SiteSage. 18) Pricing Of the 450 non web connected meters, how many different protocols/file types/file formats will be employed? A: The majority of the non web connected meters will be HOBO U12 and Pendant series and Dent ElitePro SP 19) Pricing Will all of the non web connected meters have their data downloaded manually to VEIC employee laptops? A: The intent of 2.3 is to highlight that data from a remote device can be downloaded to an intermediary medium (laptop, thumb drive, etc) before being uploaded to the database by a VEIC employee. 20) Task 1. Sub metering Data Storage How many employees will be required to VEE the interval data? A: Approximately five VEIC employees will require Admin access to the system. Please describe any limitations to your system or the impacts on pricing of having multiple users. Please describe any manual steps required to VEE data in your system. 21) Task 1. Sub metering Data Storage Is interval data currently being VEE'd at VEIC? A: Interval data VEE is currently done in an ad hoc process as part of the analytics process. Automating part of this process is desirable. 22) Task 1. Sub metering Data Storage Will master meter data be available to the platform to support VEE functionality? A: No 23) Task 2. Data Import Functionality Four devices are currently listed in 2.2, how many additional devices will the proposed system be required to support? A: The four devices listed in 2.2 are the most common devices currently used by VEIC staff. See 2.1 for questions on how the system would allow for support of additional devices. 24) Task 2. Data Import Functionality Seven devices are listed in 2.3, how many additional devices will the proposed system be required to support? A: The seven devices listed in 2.3 are the most common devices currently used by VEIC staff. See 2.1 for questions on how the system would allow for support of additional devices. 25) Task 1. Sub metering Data Storage Is the ability to VEE interval data a requirement? A: This is desirable but not required.