RESPONSE TO PRE-BID VENDOR QUERIES. SERC/HPCS/1/2014 dt

Transcription

1 RESPONSE TO PRE-BID VENDOR QUERIES SERC/HPCS/1/2014 dt This pre-bid clarification document should be read in conjunction with the tender document (SERC/HPCS/1/2014) and the changes/relaxation made are only to those clauses which relate to them. All other terms and conditions stated in the tender document, and the additional conditions stated in this pre-bid clarification are applicable as is. General Request for Site Visit: 1. We were not able to visit IISc site on Feb 4, 2014 due to the time needed for obtaining business visas and making travel arrangements. Therefore we request Your Honour to kindly approve our experts to visit site on another agreeable date. Response: Noted. Vendor requested to fix a mutually convenient date for the visit. 2. Several bidders requested for extension of submission deadline for about 2 weeks. Response: Noted. The submission deadline to the tender document is being extended to March 19, 2014, 5.00 p.m. (from March 7, 2014). 3. Non-disclosure agreement binding on bidders with regard to benchmarks and results. Response: All bidders who have taken the benchmark data (executable and/or source code and input data sets) are bound by the non-disclosure agreement and hence CANNOT use the benchmark data or the results produced for any other purpose except for this particular tender. 4. Official communication mode for the bidders regarding information concerned with this tender. Response: All bidders are herewith informed that any communication, regarding this tender, from them has to be only 1

2 through to Canvassing in any other manner by the bidder will entail disqualification. Technical specifications 1. We request IISc to look at "both" NVIDIA GPU and INTEL Xeon PHI as their choices for GPU's/Accelerators ". DELL has one of the largest cluster facilities at Texas Advanced Computing Center powering a 10 Petaflop cluster using Xeon PHI. Response: Noted. The specification for GPU-Cluster to now read as Accelerator-based cluster. The term GPU must be interpreted as Accelerator within the context of the tender document. The specification in item 7 for the GPU-cluster is modified as below: Accelerator cards Not more than 2 cards per node of Intel Xeon-Phi 5110P or 7120P or Nvidia K20 or K20x. The compute power provided by the accelerator nodes should be 100TF sustained HP Linpack performance. 2. Table 1 mentions two clusters: HPC-CPU-Cluster and HPC-GPU- Cluster. Table 2 further describes these cluster specifications asking for master and login nodes, service nodes, software environment (parallel development environment, GPU software environment etc.). Do these clusters need to be fully functional independently with each one having its own software licenses? Or do they need to be in a single pool with defined functionalities? Response: If the solution for the CPU cluster is based on x86 processor architecture then a single pool with defined functionalities is what is required. Otherwise these components can operate as independent clusters. 3. Number of concurrent users (for compilers); concurrent MPI tasks/cores (for debuggers) that need to be quoted may please be provided. Response: Number of concurrent users for compilers is 30 (thirty) and for debuggers concurrent MPI(tasks/cores) is 2048 with additional Under high performance computing storage specifications (section C of Table 2) IOR/IOZONE benchmarks is prescribed for throughput measurement. Please provide block size that needs to be used for running this benchmark. 2

3 Response: 1024KB block size can be used. 5. Pg 5, #8 #8 Architecture specific compilers - can this include GNU compilers also Response: The software stack for the proposed solution should be the same as that used for benchmarks. Any deviation is not acceptable. 6. Pg 5, #8 #8 Debuggers - Please include Parallel Environment Developer Edition (PEDE) as an option, since it is compatible with the architecture we intend to propose. Response: Technical specification indicates the kind of features required. If the vendor feels that a particular open-source / proprietary product meet the requirement, he may propose in this category with appropriate justification included. 7. Pg 5, #10 #10 File System - Can PFS be other open source FS. If Lustre, is it a OEM supported version? Response: PFS or any other open source FS can be quoted if the same has been used during the benchmark runs. For any product, since SERC is intending to buy the solution, it is expected that the bidder has the capability and competence (either by himself or by way of the OEM of the product) for all the products that have been proposed as part of the solution. 8. Pg.6 storage #1 - Especially for HPC based workloads, IBM has introduced GPFS based (software( RAID technology which offers better functionality and performance as compared to traditional hardware based RAID controllers. Besides, these also offer tremendous savings in terms of space, power and cooling, We request IISc to include file system based software RAID technologies also in the proposed specification for storage. Response: Bidder can propose this as an optional item. 9. Pg 6 storage #2 - We request IISc to include the specification of "10K rpm" disk Response: Noted. The specification changed to 3

4 6. Optional local storage for OS & s/w env. 1 X 300GB SAS or more for solutions supporting local storage on compute nodes. 10. Suggest to have Snapshot functionality which would enable IISc to have multiple point in time snapshots of a complete file system which would help in quick recovery of individual files or the complete file system from an earlier snapshot and provides protection against any data corruption that may happen. Response: This can be proposed as an optional item. 11. Suggested to have an automated hierarchical storage management functionality to move data/files from the SAS pool to the NL- SAS/SATA pool, based on predefined policies Response: This can be proposed as an optional item. 12. In page # 5, under "Master and login node redundancy" it is mentioned as "4 management/service/login nodes must be made available for redundancy and in load balancing mode". Please clarify how many master and login node redundancy required. To these need to be cold standby nodes? Response: The number covers all such nodes put together. Each node must be configured such that they can take the role of any of the functionality. They are expected to be in active redundant mode. Further the following row in the technical specification of the CPUcluster is spurious and could be omitted. 13. Service nodes 2 or more service nodes to be provided for management and monitoring of the server. Hardware configuration must be compliant to support the service management tools for the server. 13. Since latest generation AMD processors 63xx is available for quite some time now, could that be asked instead of older generation 62xx processors. This will help the institute to get latest generation processor. Response: Noted. RFP technical specification modified as below: For the CPU-Cluster 4

5 3. Processor Architecture (64-bit) Intel Xeon E5 v2 (Ivybridge) at 2.4 GHz or higher or AMD Opteron 63xx (Abu-Dhabi) at 2.4 GHz or higher, or IBM Power7 at 3.8GHz or higher or IBM PowerPC at 1.6GHz or higher For the Accelerator-cluster 3. Processor Architecture (64-bit) Intel Xeon E5 v2 (Ivybridge) at 2.4 GHz or higher or AMD Opteron 63xx (Abu-Dhabi) at 2.4 GHz or higher, or IBM Power7 at 3.8GHz or higher or IBM PowerPC at 1.6GHz or higher 14. In page # 3, under "SPECIFICATIONS AND TECHNICAL DETAILS", it is mentioned as "100 TFFLOPS coming from GPU-cluster (with CPU-GPU combination)". Please clarify 100TF sustained only from GPU cards or CPU+GPU. Response: For the Accelerator nodes, both CPU and the accelerator together contribute to the 100TF sustained capacity. 15. In page # 4, under "Master and login node redundancy" it is mentioned as "8 management/service/login nodes must be made available for redundancy and in load balancing mode". Please clarify how many master and login node redundancy required. Will these be cold standby? Response: As answered under Query 12 of this section. 16. Request to consider block size of 1MB for large sequential write. Also please clarify whether 2TB or 3TB drives to be considered. Response: Noted. 3 TB drives to be considered for the 2PB storage solution and 1024KB block size for large sequential writes. 17. Is it acceptable to put only 1 disk per node? Response: Yes, please read the modified tender specification under Query 9 of this section. 18. Memory in GPU part : 4GB per CPU core but not per GPU core Response: Yes 5

6 19. Storage: disk of 2 TB/ 3 TB are mentioned. Can we use disk with different capacity? Response: The revised specification requires only 3TB disk to be used. 20. IB: FDR latency < 2 microseconds would imply a maximum of 2 levels of switches. Can this limit be raised to 3 microseconds? Response: No. 21. Node-to-node interconnect a. Greater than 50 Gb/s b. Inter-rack > 35 Gb/s Does it mean that in the fat tree a pruning factor of 35/50 is accepted? Response: Noted. The bandwidth specification is not related to the FAT-tree pruning factor. The specifications of the tender document are final. 22. Table 2/point 6 concerning the optional local storage of 2x500GB SAS or SATA. Does this requirement mean that it is 0 or 2 disks per compute node or it is possible to have 1 x 500GB SAS or SATA per compute node? Response: Noted and modified. Please read response to Query 9 under this section. 23. Table 2/point 4 concerning the total memory >= 100TB on the first cluster of 500Tflps HPL. Is it acceptable to have heterogeneous nodes in terms of quantity of memory per node or all compute nodes should be configured with the same quantity of Memory? Response: All master and compute nodes in the solution must have identical configuration. With regard to memory configuration of each node, it should be balanced and configured for optimized performance. 6

7 Benchmarks 1. For technical pre-qualification of bidders, IISC to consider any 3 out of 4 benchmark (presently all 4 benchmark results are required as pre-qualifier), this will help bidders to complete the benchmark runs with available time on hand or request IISc to extend the due date for submission. Response: As stated in the tender document, submission of performance results for all benchmarks is mandatory. 2. Bidders need to be allowed to utilize their OEM's/Technology provider's facility to run these benchmarks with IISC reserving the right to solicit full disclosure reports as necessary. Response: The bidder is permitted to use only his/hpc OEM s facility for the benchmark runs. 3. Point number 4 under benchmark Performance Criteria allows vendors to use one generation older processor. Is it allowed to use a processor running at a different clock speed (either of the same generation as proposed or one generation older) for running the benchmark? Is performance projections to this effect (clock speed difference between proposed and benchmark system) allowed? Response: One generation older processor and/or clock speeds in the range of 2.2 GHz to 2.8 GHz for Intel/AMD processors or 3.5 GHz to 4.2 GHz for Power 7, or 1.4 GHz to 1.8 GHz PowerPC processors can be used. In addition the benchmark system used should not assume HPL efficiency lower than 80%. Thus, if the published HPL efficiency of the benchmark system is X%, then the system used for benchmarking should have a peak performance in the range of (400*100/max(X, 80)) to (500*100/max(X, 80)). In all cases no projections of benchmark numbers will be permitted. Whatever results that are submitted as part of the tender response, the same will be considered as is for evaluation. 4. The RFP asks for the best execution time for CESM. Could you please confirm whether we need to report the execution time on a 500TF size system OR the best performance achieved within the 500 TF system? If the application stops scaling before 500TF size system best performance may be at a lower size system. 7

8 Response: The best execution time achieved on a system with TF sustained HPL performance must be submitted as a part of the tender response and is a mandatory requirement. 5. Are the source code and input data for CESM, Zeus and HiFun same as earlier RFP? Response: The vendor must collect the BM associated source/binary and input data from SERC and should use the same for the benchmarking. Modifications neither to the source code nor to the input data set are permitted. 6. For HiFun the problem is fixed for cores. At one place it says on a 500TFLOPS system utilizing as many cores as appropriate and at one another place it says utilizing a machine topology exercising all the cores in each of the nodes. Please provide some clarity. Do we have to use all the nodes in 500TF and all the cores in those nodes? It could be more than cores Response: HiFun requires processes and the bidder is expected to use cores for this run on a machine. However, these cores must be so selected that all cores on a node contribute to the machine topology for the benchmark. 7. In the earlier RFP, for Zeus the benchmark problem size was 1152x1024x1024 with a tile size of 36x32x16 or core nodes (around 300 Tflops). In this RFP, the problem size is now 1664x1536x1536 with a tile size of 32x32x24 or core nodes. This means that the ratio between problem size and tile size is not consistent with previous releases of this benchmark by SERC. We would like to confirm that this problem should be run with the input lines: &mpitop ntiles(1)=32,ntiles(2)=32,ntiles(3)=24,periodic=3*.true. / &ggen1 nbl=1664,x1min=0.0,x1max=1.2344e+23,igrid=1,x1rat=1.,lgrid=.true. / &ggen2 nbl=1536,x2min=0.0,x2max=1.2344e+23,igrid=1,x2rat=1.,lgrid=.true. / &ggen3 nbl=1536,x3min=0.0,x3max=1.2344e+23,igrid=1,x3rat=1.,lgrid=.true. / (or scaled up to the 500TF level) If the problem size and tile size are as stated, then it seems that the number of zones has to be increased. Is there any restriction or guidance on how this should be done? For example, if IZONES and JZONES are doubled, must KZONES be doubled as well? Response: The specification for the Zeus benchmark described in the tender document is being corrected as follows: 8

9 2 Zeus : For the given Zeus code and zmp_inp input file use the following input parameters for the 500TFLOPS run: Problem size = 1664 X 1536X And Tiles=32 X 32 X 24. The number of processes, specified in terms of the tile size could be scaled to the number of cores in the 500 TFLOPS system. Keeping the problem size intact, the tile size can be changed to suite the number of processes to be used depending on the system configuration used for the benchmark run. 8. Pg 4, #3 - #3 - If benchmarks were not done on the proposed SKU, are we allowed to extrapolate. If benchmarks were done on Sandybridge, are we allowed to propose ivybridge with adequate considerations taken for benchmarks Response: No extrapolation of benchmark performance is allowed. See also the response for Query 3 of this section (Benchmark). 9. Pg 8, #4 If benchmarking is performed on say a cluster utilizing 8 core Sandybridge, is there any guidelines for proposing the configuration on a 10 core ivybridge architecture. Can we take the same number of cores? Response: Noted. Specifications stated in the Tender documents and the clarifications provided in this Prebid-Clarification already address this. 10. Request IISC to nominate its team to co-ordinate regarding the benchmark queries / clarification that we may require while executing the programs. Response: Bidders can forward benchmark specific questions, if any, to office@serc.iisc.ernet.in. Based on the nature of question, the benchmark team will be asked to respond. 11. Availability of 500TF system is a challenge, kindly allow benchmark execution on 300TF system and provide application benchmark parameters specifically problem size to run on 300TF system. Response: The benchmark system size specified in the tender document along-with the clarifications provided in this document is final. 9

10 12. Considering the availability of time on hand, we request that for technical pre-qualification of bidders, IISC to consider any 3 out of 4 benchmark (presently all 4 benchmark results are required as pre-qualifier), this will help to complete the benchmark runs with available time on hand. Response: Submission of benchmark results, as stated in the tender document, for all four benchmarks is mandatory. 13. Please clarify what precision to consider, also 3MX3M requires a huge RAM, please let us know that what would be problem size for 300TF Response: The scalapackmatrixinversion program is provided as part of the benchmark data and it is built to run in double precision. No change to source code and input data size is permitted. 14. Due to large size system required for benchmarking, one might need to look at cluster system from CPU manufactures like Intel/AMD, we request IISc to relax this clause. Response: Noted. Conditions stipulated in the tender document are final. Also see the clarification provided in response to Query 2 in this section (Benchmark). 15. Requesting to provide benchmark figures expected for each of the applications. Response: Bidders are expected to give the best achievable execution times for their solutions. Also see the clarification provided in response to Query 3 and 4 in this section (Benchmark). 16. Can the performance for the four programs be an extrapolation to a system with 500 TFLOPS from measurements on a system with a sustained HPL performance of maximum 45 TFLOPS and with a maximum of 2048 cores? Response: See the clarification provided in response to Query 3 and Query 4 in this section (Benchmark). 17. To what extent the benchmark system can be different from the proposed system? 10

11 Response: The node hardware form factor including the availability of local disk of the benchmark system need not match the proposed system. However, this is subject to meeting all the conditions stated in the tender document and this Prebid- Clarification document. Infrastructure 1. We also request Your Honour to kindly provide a Plan Map of the site(to scale 1 :100) and also indicating various existing infrastructure details i.e. UPS, Chillers, etc. Response: All details will be conveyed to the bidder during the site visit. 2. Request for considering Lower PUE (due to use of hot water cooling): Cl Response: Not possible. Same as stated in the tender document. 3. Clause of Tender specifications fix the PUE at 1.25 for all bidders and does not allow the important advantage of use of hot water cooled computing nodes, i.e. energy efficiency. Response: Noted. Conditions stated in the tender document are final. 4. We request Your Honour to consider the actual PUE for each bidder and consider in the evaluation. The most energy efficient may be given 20 marks and others prorate percentage. Response: Noted. Conditions stated in the tender document are final. 5. Details of existing UPS, Chillers, Battery, DG sets etc : Please provide complete details, technical specifications and characteristics of existing infrastructure, electrical diagrams, i.e. all chillers, UPS, DG sets, Battery, etc. We presume that adequate fire protection and security systems are already provided. Response: All details will be conveyed to the bidder during the site visit. 6. Under infrastructure requirement table says it is desirable to limit power less than 1000 KVA and cooling less than 100 TR. A 11

12 system with 1000 KVA power rating would need larger TR cooling than 100TR. Would it be fine if additional cooling capacity is included in the proposal? Response: What is available for the HPC system in SERC as spare capacity is 2 X 500KVA UPS and 2 X 50 TR, total of 100 TR Chiller as given in the RFP. Beyond this, additional requirement has to be provided and included as part of the turn-key solution. In addition, the bidder has to give complete details and justification of the sizing of the chiller and the UPS. Modified technical specification template is being released with this document. 7. For chillers it is mentioned N+N in proposed chiller column (refer attached excel) where as in the main document it mentions N+1 - What is the redundancy required for chillers. Response: It is N + 1 in terms of number of chiller units. 8. In case of UPS calculation, we will consider 2 x N x 0.8 KVA considering 80% loading in UPS when N is in operation (N is load). Can we take the power factor of the server power supply for converting KVA to KW - eg is the power factor for platinum plus rated power supply. Response: If the total IT load is X KVA, then UPS sizing should be 2 x (X-400)*1.25 KVA. The proposed UPS should have an output power factor of at least 0.9 and an efficiency of 90%. In addition, the bidder has to propose UPS system which falls atleast under the category of Gold (in terms of services) to adhere to the SLA requirements stated in the tender document and to give complete details and justification of the sizing of the UPS. Modified technical specification template is being released with this document. 9. Is it okay to propose additional stand-alone DG system if in case additional DG power than 1500 KVA is required. If not, the entire Synchronization panel needs to be changed / modified which will be a difficult task. Response: Yes, noted. No change to the specification in the tender document. In addition, the bidder has to give complete details and justification of the sizing of the DG. 12

13 10. Is it okay to provide UPS with transformer or is it preferred to have transformer less UPS with external isolation transformer in the PDU Response: Built-in isolation transformer with both side IGBT technology is to be quoted. It is also to be noted that the battery backup required is to be sized to meet the tender requirements using 2 (two) volt batteries. 11. PUE mentioned - is it Catagory 1 or Catagory 2. which is considered as annual average (as per definition) Response: Category The DG set considered is only N capacity, please confirm that there is no need to increase it to N+1. Response: Yes, it is of N capacity. 13. More technical specs required on chillers / pumps - requirement of VFD drives, chiller plant Manager etc Response: The bidder has to understand the requirements and appropriately design an optimal solution to meet the stated requirements and integrate the proposed infrastructure with the existing one. 14. Is IISc expecting the bidder to provide the Technical solution and requirement for Infrastructure support for HPC or should the bidder also own the supply, installation, commissioning, maintenance of the Data Center infrastructure Response: Yes, what is required is a total turn-key solution. 15. Cooling requirement of 100 TR is not proportional to power requirement given. Please clarify on how 100 TR cooling requirement is arrived. Response: Answered in Query 6 of this section. 16. Any Civil Works if required for Chiller, New UPS Room Area has to be considered in our Scope or will be done by SERC? Response: All civil work necessitated for this solution will be taken up by SERC. 13

14 17. PSU Redundancy : a. For redundancy 1+1 should the plugs also be redundant? b. This redundancy is needed just for the compute part or is it needed for all the equipment? Response: Regarding the redundancy requirement for the infrastructure part, the solution supplied has to have the required redundancy for the UPS, Batteries and the Chiller units. 18. Chiller capacity of 100TR : is 1 TR =12 kbtu/h=3,5 kwh? which means we can use 350 kw? Response: Spare capacity of 50 TR is available and if the total requirement of chiller is more than 50 TR, additional capacity has to be quoted, the details of which are given in the modified technical specification excel sheet. 19. How must the IT-power be calculated? Response: The IT power required is calculated as the total power required for the entire HPC system including the CPU-cluster, Accelerator-cluster and Storage while running the HP-Linpack program on the CPU and the Accelerator Cluster with turbomode ON. This power requirement should corroborate with the sizing of the infrastructure requirement including the UPS, Chiller and the DG. Technical Evaluation 1. Since hot water cooled nodes allow the turbo-boost to be run permanently, we request Your Honour to kindly consider the advantages of turbo-boost mode in the technical evaluation. Response: Noted. Conditions stated in the tender document are final. 2. Evaluation provides a weightage for number of systems in the Top 500 list which favors large corporation(s). This weightage may be removed completely (since qualification criteria for participation to the tender as listed in Bidders' Eligibility Criteria Clause 1). Such a criterion is seen by us for the first time and gives a 5%+ advantage to a large corporate and is not in the best interests of the customer. Please consider. 14

15 Response: Noted. Conditions stated in the tender document (reference to section 3.3.5) are relaxed to consider all solutions with HPLinpack performance of 400TF or higher. Other conditions remain as stated in the tender document. 3. Pg Since the evaluation mechanism has a methodology for awarding more marks to the solution utilizing least power, we request that this clause be deleted or at least increase the factor to 2.5. This clause limits the options that we can propose, which other than the power consumption, can offer superior features to IISc. Response: Noted. Conditions stated in the tender document are final. 4. It appears the scoring system gives more focus on UPS power required in KVA. ROC shall be based on 1.25 * UPS power in KVA. Lowest will be given 20 points. Tenderer with ROC more than 1.5 x ROC minimum shall be rejected. Please clarify whether our understanding is correct. Response: Yes. 5. Reference to operational cost in the evaluation criteria - Operational Cost - Scope of services covered under Operational cost, Baseline parameters considered for operational cost Response: The PUE mentioned in the RFP, which is uniform across all the bidders is estimated taking into account the operational cost. 6. Since this clause (Clause 2 of Bidder s eligibility criteria) is to ascertain the capability of the OEM/ SI, we request you to also consider the number of implementations carried out by the OEM, which are listed by Top 500. Response: This clause is related to the bidder s capability aspect and is being relaxed, for enabling maximum participation, to read as below: The bidder must have a proven record of maintaining and managing at least one system having 200TF (peak performance) or higher for a minimum period of six months. Appropriate documentary evidence with a letter from the customer reporting 15

16 the details of the maintenance/management responsibilities and the performance of the bidder should also be included in the technical bid of the proposal. 7. Concerning the criteria on number of Systems in the Top 500 it is written: For each additional system, 0.4 marks will be given, subject to a maximum of 8 additional marks Does this refer to any systems in the Top 500 or is it restricted to systems above 500 Tflps? Response: It refers to systems (sustained HPL of 400 TF or above) which are currently listed in the Top500 list (of Nov. 2013) as well as systems which are currently operational but not in the Top500 (as of Nov. 2013) list. Based on the representation made in the prebid-clarification meeting, it is revised as follows: For each additional system having a sustained HPL performance of 400 TFLOPS and above, 0.4 marks will be given, subject to a maximum of 8 additional marks. For systems not listed in the Top500 list (of Nov. 2013), documentary evidence must be enclosed as a part of the technical bid. Terms and conditions 1. We request Your Honour to kindly study and approve a project period of 20 weeks. This will allow sufficient time for in-plant inspection of critical components, processors etc., i.e. project specific quality inspections, etc. and will benefit the customer. Response: Noted. Conditions stated in the tender document are final. 2. Two weeks is a very short duration to conduct all tests, especially for large system such as this. Request the time be extended to 4 weeks. Response: Noted. Conditions stated in the tender document are final. 3. Warranty and AMC charges - We understand that all proposed equipment needs to be proposed with a warranty of three years and subsequent AMC charges for 4th and 5th year. Would AMC charges be included in the evaluation methodology? Can it be excluded? 16

17 Response: The TCO calculation, as stated in the tender document is for the complete solution, 3 years of warranty charges and two years of post warranty AMC charges put together. 4. Under Clause 10/97-ce, ED is exempted and not Customs Duty - Please clarify. Response: The Institute is eligible for the excise duty as well as the customs duty wherever applicable. 5. Clarification on the Condition 4 under Acceptance Criteria of the tender document whether the modified solution should include the required infrastructure to support it as part of the installation. Response: Yes. The modified condition to read as follows: If the bidder fails to demonstrate any of the above requirement, the technical team of the Institute will come out with appropriate configuration that should have been quoted, which would have met the HPL ratings and the benchmarks performance, and the bidder has to supply, install and maintain a solution twice that capacity (including the required infrastructure namely, UPS, Chiller and Diesel capacity along-with the HPCS solution) at no extra cost to the Institute. 17