| ID |
Date |
Author |
Category |
Subject |
|
64
|
Tue Feb 4 19:52:22 2020 |
Dinko Ferencek | Cold box tests | Blue coldbox setup for reception tests commissioned | Today I continued where Matej left off with commissioning the blue coldbox setup for reception tests. All the necessary software (pXar and elComandante) Matej already installed (just had to add the BB2 section in testParameters.dat). What was a problem in communication with Keithley (Model 2400) which was fixed after changing the BAUD rate in elComandante/keithleyClient/keithleyInterface.py to 19200 (value set in Keithley's communications settings) and changing FLOW-CRTL in Keithley's communication settings to XON-XOFF.
The reception test was successfully run for modules 1545, 1547, 1548, 1549, and 1550 after cap gluing. |
|
65
|
Wed Feb 5 16:07:03 2020 |
Dinko Ferencek | Module assembly | Module assembly spreadsheet migrated to Google Sheets | Excel file stored on CERNBox containing the module assembly spreadsheet has been replaced with a Google Sheet https://docs.google.com/spreadsheets/d/12m2ESCLPH5AyWuuYV4KV1c5O_NEy7uP5PQKzVsdr3wQ/edit?usp=sharing. This change should make common editing of the spreadsheet easier.
The new spreadsheet has also been linked from the main L1 replacement web page http://cms.web.psi.ch/L1Replacement/ |
|
66
|
Wed Feb 5 16:18:30 2020 |
Dinko Ferencek | Module transfer | Module location tracking added to the assembly spreadsheet | | To keep track of the module location, two columns were added to the module assembly spreadsheet https://docs.google.com/spreadsheets/d/12m2ESCLPH5AyWuuYV4KV1c5O_NEy7uP5PQKzVsdr3wQ/edit?usp=sharing that contain the dates when a given module is shipped to ETHZ and when it is returned to PSI. |
|
67
|
Thu Feb 6 00:45:25 2020 |
Dinko Ferencek | Cold box tests | Lost at least one Peltier in the coldbox | During an overnight (Feb. 4-5) FullQualification run with modules 1545, 1547, 1548, and 1549 the coldbox lost the ability to maintain the temperature at -20 C. As can be seen from the attached temperature log, the problems already started during the temperature cycles where the last 2 cycles were noticeably longer than the first 3 and finally during the second Fulltests at -20 C the temperature started rising and stabilized around -10 C. Based on these observations it looked like at least one Peltier stopped working.
Silvan took apart the coldbox and will install new Peltiers. Based on measurements Andrey did with the old Peltiers it indeed looks like one Peltier stopped working but the remaining 3 also appear to be at different stages of degradation. |
|
68
|
Thu Feb 6 17:09:50 2020 |
Dinko Ferencek | Module assembly | Gluing stamp improvements | | Silvan has made additional grooves on the gluing stamp which now sits stably on top of the module without any rocking motion and also allows better glue distribution to all the HDI components that are supposed to receive the glue. |
|
69
|
Thu Feb 6 17:19:12 2020 |
Dinko Ferencek | Reception test | RT for 6 modules: 1551, 1552, 1553, 1554, 1555, 1556 | Today reception test was run for 6 modules and looks OK for all module. The modules were graded as follows:
1551: A
1552: A
1553: B (Electrical grade B)
1554: A
1555: B (IV grade B)
1556: B (IV grade B)
Protective caps were glued to these modules. |
|
70
|
Thu Feb 6 17:28:23 2020 |
Dinko Ferencek | Cold box tests | New Peltiers installed | Today new Peltiers were installed and tested. Unfortunately, we had trouble reaching -20 C with this new set of Peltiers (they are of different type than the ones that were taken out). After better insulation ofthe sample compartment and lowering the chiller temperature from 11 to 6 C, the coldbox was able to reach close to -19.8 C after about half an hour
After lowering the chiller temperature to 5 C, the coldbox was able to reach -20 C
and hold it for about 1 hour when the test was stopped
The plan is to order a new set of Peltiers that are better suited for our use case and in the meantime we will use the newly installed ones with the chiller temperature set lower.
IMPORTANT: Pipes inside the coldbox were left uninsulated. For now this is OK because of the low relative humidity in winter months but could lead to condensation in warmer part of the year when the relative humidity increases. |
|
71
|
Fri Feb 7 19:40:27 2020 |
Dinko Ferencek | Reception test | RT for 6 modules: 1557, 1558, 1559, 1560, 1561, 1562 | Reception test is done and all 6 modules were graded A.
Protective caps were glued to 4 modules: 1557, 1558, 1559, 1560 |
|
72
|
Fri Feb 7 20:35:44 2020 |
Dinko Ferencek | Full test | FT for 4 modules: 1545, 1547, 1548, 1549 |
1545: C/C/C (-20/-20/+10). Problematic ROC 14: mean noise >320e, many dead trimbits, wide trimmed Vcal threshold distribution. IV: 0.20 uA at +10 C, slope 1.09
1547: B/B/B (-20/-20/+10). Electrical grade is B due to mean noise >200e for some ROCs. IV: 0.41 uA at +10 C, slope 1.08
1548: B/B/B (-20/-20/+10). Electrical grade is B due to mean noise >200e for some ROCs. IV: 0.13 uA at +10 C, slope 1.10
1549: B/B/B (-20/-20/+10). Electrical grade is B due to mean noise >200e for some ROCs. IV: 0.33 uA at +10 C, slope 1.12
Chiller operational procedures:
Chiller temperature was initially set to 4 C and the coldbox was able to reach -19.7 C which is within the require 0.5 C margin and the Fulltest could start. About 1 hours into the Fulltest, it was observed that the coldbox temperature rose to 19.0 C. At that point the chiller temperature was lowered to 3.5 C and the coldbox was able to lower and keep the temperature in the -19.2 to -19.3 C range. Once the test at -20 C were done, for the Fulltest and IV at +10 C the chiller temperature was initially raised to +11 C and after about one hour (midway through the Fulltest), it was lowered to +10 C because at one point the coolant temperature rose to +13 C. The temperature history plot is attached. |
|
73
|
Sat Feb 8 20:48:11 2020 |
Dinko Ferencek | Full test | FT for 4 modules: 1550, 1551, 1552, 1553 |
1550: B/B/B (-20/-20/+10). Electrical grade is B due to mean noise >200e for some ROCs. IV: 0.45 uA at +10 C, slope 1.09
1551: C/C/C (-20/-20/+10). Mean noise >200e for some ROCs, dead trimbits in ROC 13, but trimmed Vcal threshold distribution and distribution of trim bits look OK. IV: 0.33 uA at +10 C, slope 1.09
==> Manually regraded to B
1552: B/B/B (-20/-20/+10). Electrical grade is B due to mean noise >200e for some ROCs. IV: 0.23 uA at +10 C, slope 1.10
1553: B/B/B (-20/-20/+10). Electrical grade is B due to mean noise >200e for some ROCs, double-column readout problem in ROC 11 (see attached plot). IV: 0.41 uA at +10 C, slope 1.70
Chiller operational procedures:
Chiller temperature set and kept at 3.5 C throughout the entire test. This was done after observing that during tests at +10 C and with the chiller left at +3.5 C, the Peltiers were mostly inactive and there was no excessive heating occurring. This seems to suggest that the coolant temperature at +3.5 C is a good balance against the ambient temperature and heat produced by working modules to keep the temperature at +10 C. The temperature history plot is attached. |
|
74
|
Sun Feb 9 10:52:17 2020 |
Dinko Ferencek | Full test | FT for 4 modules: 1539, 1554, 1555, 1556 |
1539: B/B/B (-20/-20/+10). Electrical grade is B due to mean noise >200e for some ROCs. IV: 0.17 uA at +10 C, slope 1.08
1554: B/B/B (-20/-20/+10). Electrical grade is B due to mean noise >200e for some ROCs. IV: 0.44 uA at +10 C, slope 1.80
1555: B/B/B (-20/-20/+10). Electrical grade is B due to mean noise >200e for some ROCs. IV: 0.99 uA at +10 C, slope 1.30
1556: B/B/B (-20/-20/+10). Electrical grade is B due to mean noise >200e for some ROCs. IV: 0.87 uA at +10 C, slope 1.67 |
|
75
|
Sun Feb 9 12:05:47 2020 |
Dinko Ferencek | Module assembly | Cap gluing | | Today protective caps have been glued to modules 1561 and 1562. |
|
76
|
Sun Feb 9 18:36:13 2020 |
Dinko Ferencek | Module grading | Manual grading | The procedure for manual grading is described in https://github.com/psi46/MoReWeb/pull/120.
In short, inside the test folder it is necessary to place a text file called grade.txt that contains just one number representing the manual grade:
1=A
2=B
3=C
Addendum:
It appears that manual grading does not work properly if one reruns MoReWeb with grade.txt added. It is necessary to first delete any entries related to the FullQualification using the following command
python Controller -d
after which you need to specify for which module you want to delete entries, e.g. M1537. Once done, you need to run MoReWeb for this module, e.g.
python Controller -m M1537 |
|
77
|
Sun Feb 9 19:34:55 2020 |
Dinko Ferencek | Full test | FT for 4 modules: 1557, 1558, 1559, 1560 |
1557: C/C/C (-20/-20/+10). Mean noise >200e for some ROCs, dead trimbits, failed trimming and mean noise >300e in ROC 4. IV: 0.20 uA at +10 C, slope 1.08
1558: C/C/C (-20/-20/+10). Mean noise >200e for some ROCs, dead trimbits in ROCs 2 and 12, mean noise just above 300e for ROC 2, possibly C* module. IV: 0.27 uA at +10 C, slope 1.07
1559: B/B/B (-20/-20/+10). Electrical grade is B due to mean noise >200e for some ROCs. IV: 0.23 uA at +10 C, slope 1.09
1560: B/B/B (-20/-20/+10). Electrical grade is B due to mean noise >200e for some ROCs. IV: 0.24 uA at +10 C, slope 1.09 |
|
78
|
Tue Feb 11 10:25:32 2020 |
Dinko Ferencek | Reception test | RT for 6 modules: 1563, 1564, 1565, 1566, 1567, 1568 | Feb. 10
1563: Grade C, ROCs 8 and 10 not programmable, no obvious problems with wire bonds
1564: Grade B, I > 2 uA (2.17 uA)
Feb. 11
1565: Grade A
1566: Grade A
1567: Grade C, no data from ROCs 8-11, looks like a problem with one TBM1 core
1568: Grade A |
|
79
|
Tue Feb 11 17:56:06 2020 |
Dinko Ferencek | Cold box tests | Fulltest failure - psiAgente | During today's full qualification the 2nd fulltest@-20 for TB1 (M1562) failed
psiAgente: self.Testboards[1] failed- the following boards are busy: TB0(DTB_WRBSJ9:M1561), TB2(DTB_WRE1O5:M1564)
This looks similar to the report made on Jan. 30.
Looking at the pXar terminal window I managed to catch the following output
Thread 1 (Thread 0x7fe20f554b00 (LWP 7987)):
#0 0x00007fe20c4e90cb in __GI___waitpid (pid=17338, stat_loc=stat_loc
entry=0x7ffe23b71480, options=options
entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
#1 0x00007fe20c461fbb in do_system (line=<optimized out>) at ../sysdeps/posix/system.c:148
#2 0x00007fe20e998172 in TUnixSystem::Exec (shellcmd=<optimized out>, this=0x1fa2570) at /home/l_tester/root/core/unix/src/TUnixSystem.cxx:2119
#3 TUnixSystem::StackTrace (this=0x1fa2570) at /home/l_tester/root/core/unix/src/TUnixSystem.cxx:2413
#4 0x00007fe20e99aa43 in TUnixSystem::DispatchSignals (this=0x1fa2570, sig=kSigSegmentationViolation) at /home/l_tester/root/core/unix/src/TUnixSystem.cxx:3644
#5 <signal handler called>
#6 0x00007fe20d5d6794 in PixTestTrim::trimTest (this=this
entry=0xcfe82f0) at /home/l_tester/L1_SW/pxar/tests/PixTestTrim.cc:359
#7 0x00007fe20d5da500 in PixTestTrim::doTest (this=0xcfe82f0) at /home/l_tester/L1_SW/pxar/tests/PixTestTrim.cc:132
#8 0x000000000040a3d0 in main (argc=<optimized out>, argv=0x7ffe23b7b828) at /home/l_tester/L1_SW/pxar/main/pXar.cc:376
===========================================================
The lines below might hint at the cause of the crash.
You may get help by asking at the ROOT forum http://root.cern.ch/forum
Only if you are really convinced it is a bug in ROOT then please submit a
report at http://root.cern.ch/bugs Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.
===========================================================
#6 0x00007fe20d5d6794 in PixTestTrim::trimTest (this=this
entry=0xcfe82f0) at /home/l_tester/L1_SW/pxar/tests/PixTestTrim.cc:359
#7 0x00007fe20d5da500 in PixTestTrim::doTest (this=0xcfe82f0) at /home/l_tester/L1_SW/pxar/tests/PixTestTrim.cc:132
#8 0x000000000040a3d0 in main (argc=<optimized out>, argv=0x7ffe23b7b828) at /home/l_tester/L1_SW/pxar/main/pXar.cc:376
===========================================================
However, looking at the relevant section of the code in tests/PixTestTrim.cc
356 for (unsigned int iroc = 0; iroc < rocIds.size(); ++iroc) {
357 for (int ix = 0; ix < 52; ++ix) {
358 for (int iy = 0; iy < 80; ++iy) {
359 if (thr2[iroc]->GetBinContent(1+ix,1+iy) > initialCorrectionThresholdMax - 2) {
360 thr2[iroc]->SetBinContent(1+ix,1+iy,0);
361 }
362 }
363 }
364 }
there is nothing special there, just getting the histogram bin content. Could this be some random memory corruption issue? |
|
80
|
Wed Feb 12 10:43:04 2020 |
Dinko Ferencek | Module transfer | 9 modules sent to ETH: 1539, 1545, 1547, 1548, 1549, 1550, 1551, 1552, 1553 | Yesterday the following modules were sent to ETH:
1539, 1545, 1547, 1548, 1549, 1550, 1551, 1552, 1553
The FullQualification tar files for these modules have been copied to the common CERNBox folder shared with ETH and the transfer dates have been added to the assembly spreadsheet. |
|
81
|
Wed Feb 12 11:15:04 2020 |
Dinko Ferencek | Module grading | Problem with dead trimbits understood | It was noticed that some modules are graded C because of typically just one ROC having a large number of dead trimbits. One example is ROC 10 in M1537
At the same time, for this same ROC the Vcal Threshold Trimmed distribution looks fine
as well as the distribution of trim bits
It turns out that the trim bit test is failing in this and other similar cases because of the tornado plot that is shifted up more than is typical
In the trim bit test code, the Vthrcomp was raised from 35 to 50 but for cases like this one, this is still not high enough. We therefore further increased the value of Vthrcomp to 70.
Grading procedure:
We will manually regrade to B all those modules graded as C due to dead trimbits provided the Vcal Threshold Trimmed distribution looks fine, the number of trimming problems is below the threshold for grade C (167), the distribution of trim bits looks reasonable, and they are not graded C for any other reason.
Side remark:
It should be noted that the Trim Bit Test distribution and the way the number of dead trimbits is counted will not always catch cases when the trim bit test algorithm failed. For instance, in the MoReWeb output for ROC 10 in M1537 one does not immediately see that there are many underflow entries
which arise from the fact that the untrimmed threshold used in the test is too might (Vthrcomp value too low) and Vcal that passes the threshold is not found and left at the default value of 0
This problem of not spotting the algorithmic failure is particularly severe in the case of ROC 2 of the same module M1537 where it goes completely unnoticed in the summary table
it is very hard to spot from the plot (because there is no statistics box showing the underflow)
but the problem is there for a large number of pixels
Even from the tornado plot one would not expect problems
but this tornado is for one particular pixel (column 12, row 22) and there is no guarantee that for other pixels the trim bit test won't fail. |
|
82
|
Wed Feb 12 15:40:13 2020 |
Dinko Ferencek | Reception test | RT for 3 modules: 1569, 1570, 1571; 1572 bad | 1569: Grade A
1570: Grade A
1571: Grade B, I > 2 uA (3.09 uA)
1572 from the same batch of 4 assembled modules was not programmable and was not run through the reception test. |
|
83
|
Thu Feb 13 21:44:28 2020 |
Dinko Ferencek | Reception test | RT for 6 modules: 1573, 1574, 1575, 1576, 1577, 1578 | 1573: Grade A
1574: Grade B, I(150)/I(100) > 2 (4.63)
1575: Grade C, problem with one TBM core
1576: Grade A
1577: Grade A
1578: Grade A |
|