CMS Pixel Detector Miscellaneous
Phase 1 Phase 2
Layer 1 Replacement Layers 2-4
  Layer 1 Replacement Elog, Page 6 of 16  Not logged in ELOG logo
ID Date Author Categorydown Subject
  309   Mon Jan 25 13:03:22 2021 Dinko FerencekPOSPOS configuration files created
The output POS configuration files has '_Bpix_' instead of '_BPix_' in their names. The culprit was identified to be the C3 cell in the 'POS' sheet of Module_bookkeeping-L1_2020 Google spreadsheet which contained 'Bpix' instead of 'BPix' which was messing up the file names. This has been fixed now and the configuration files regenerated.

The WBC values were also changed to 164 for all modules using the following commands

cd /home/l_tester/L1_SW/pxar2POS/
./pxar2POS.py --do "dac:set:WBC:164" -o /home/l_tester/L1_DATA/POS_files/Configuration_files/ -i 1

This created a new set of configuration files with ID 2 in /home/l_tester/L1_DATA/POS_files/Configuration_files/.

The WBC values stored in ID 1 were taken from the pXar dacParameters*_C*.dat files and the above procedure makes a copy of the ID 1 files and overwrites the WBC values.
  4   Tue Aug 6 16:06:45 2019 Matej RoguljicOtherVsh and ctrlreg for v4 chips
The recommended default value for Vsh is 8, but the current version of pXar has it as 30. One should remember this when making new configuration folders from mkConfig. The recommended value of ctrlreg is 17.
  8   Tue Sep 10 15:18:20 2019 Matej RoguljicOtherModules 1504, 1505, 1520 irradiation report
Modules 1504, 1505 and 1520 were taken to Zagreb for irradiation on 13.08.19. The goal was to check the v4 behavior after irradiation. They were irradiated to 1.2 MGy and returned to PSI on 9th of September. Upon testing them at PSI, they all had issues. ROCs on M1504 and M1520 were not programmable at all, changing Vana had no effect. Vana could be set on M1505 while targeting Iana = 28 mA/ROC, however, no timing could be found for it and no working pixel could be found. Andrey and Matej took the modules under the microscope and saw a "greenish" deposits on HDI metal pads of unknown origin. There was also a bit of liquid near the HDI ID on M1520, but not on others. The residue could be shorting some pads causing issues on the module. It is still unclear whether the residue comes from the HDI, cap or is it introduced during irradiation by something in Zagreb.
  10   Fri Sep 13 15:06:51 2019 Andrey StarodumovOtherModules 1504, 1505, 1520 irradiation report
It was discovered that these modules were stored in Zagreb in the climatic lab where T was about +22C. These modules have been transported from Co60 irradiation facility to the lab on open air with T>30C and RH>70%. Water in the air under the cap condensated on the surface of HDIs and diluted residuals (from soldering, passivation etc), that after remains liquid or crystallised.
The irradiation itself does not course any damage. This is also confirmed by the fact that after two previous irradiations in Jan and Jul 2019 of modules and HDIs, samples remained in a good shape without any residuals on HSI surfaces. These samples have been kept in an office where T and RH were similar to the outside and not in the climatic lab.

We consider the the case is understood and closed.
  12   Thu Sep 19 00:52:24 2019 Dinko FerencekOtherModules 1504, 1505, 1520 irradiation report

Andrey Starodumov wrote:
It was discovered that these modules were stored in Zagreb in the climatic lab where T was about +22C. These modules have been transported from Co60 irradiation facility to the lab on open air with T>30C and RH>70%. Water in the air under the cap condensated on the surface of HDIs and diluted residuals (from soldering, passivation etc), that after remains liquid or crystallised.
The irradiation itself does not course any damage. This is also confirmed by the fact that after two previous irradiations in Jan and Jul 2019 of modules and HDIs, samples remained in a good shape without any residuals on HSI surfaces. These samples have been kept in an office where T and RH were similar to the outside and not in the climatic lab.

We consider the the case is understood and closed.


Just to clarify. The modules were not transported from Co60 the irradiation facility to the lab in open air but inside a closed Petri dish. Otherwise, there would be no risk of water condensation if the air surrounding modules was allowed to quickly mix with the air-conditioned lab air. Here the problem arose from the fact that it was not only the modules that were brought inside the lab but they were brought inside a pocket of the outside air. A closed Petri dish is not airtight but it significantly reduces mixing of the air inside the Petri dish with the surrounding lab air making it slower than the rate at which the Petri dish and the module inside it were cooling down once brought inside the lab. This could have led to water condensation if the pocket of air trapped inside the Petri dish was warm and humid and had a dew point above the lab air temperature. To prevent this from happening, the solution should be relatively simple and it would be to open the Petri dish and uncover modules before bringing them inside the lab. That way the exchange of air will be faster and the risk of condensation will be basically gone because a warm module will be quickly surrounded by the lab air which will not condense on a warmer surface.

However, there is an additional twist in this particular case. On Aug. 22, when the modules were brought in the lab, there was a thunderstorm in Zagreb in the early afternoon (https://www.zagreb.info/aktualno/zagreb-je-zahvatila-oluja-munje-i-jaka-kisa-nad-vecim-dijelom-grada/227156) with temperature around 21.5 C and RH around 75% at the time the modules were transported (around 15:20), and the whole day was relatively fresh and humid. The outside air on that day would definitely not lead to water condensation in the lab. However, before being brought in the lab, the modules were sitting in a room in the building where the Co60 irradiation facility is located so the air inside the Petri dish was likely similar to the air inside that room (the modules were sitting there for a while and there was enough time for the air temperature and humidity to equalize) and there was not much time to mix with the outside when being transported from one building to another. Unfortunately, there are no measurements of the air temperature and humidity in that room. However, it is worth mentioning that the previous day, Aug. 21, was not very hot and humid with midday temperature around 26 C and RH around 50%. It is therefore likely that the air inside that room and consequently inside the Petri dish was not very hot and humid, making the hypothesis of water condensation in the lab, if not improbable, certainly less likely.

Either way, more careful handling of modules will be needed.
Attachment 1: ZG-FER_temp_hum_2019.08.16.-09.16.png
ZG-FER_temp_hum_2019.08.16.-09.16.png
Attachment 2: pixellab-main-room-1.png
pixellab-main-room-1.png
  99   Wed Mar 11 15:34:57 2020 Urs LangeneggerOtherM1586: issues with MOLEX?
Module M1586 had passed the full qualification on 20/02/27. I had had to re-insert the cable in the Molex connector for it to become programmable.

On 2020/03/09, I tried to re-test M1586, but it was not programmable. Visual inspection revealed nothing to me. I did re-insert the cable once again, but this time this did not help.

Maybe one should try again re-inserting the cable.

Maybe these issues are an indication that the module (MOLEX) is flaky.
  100   Wed Mar 11 16:48:10 2020 Andrey StarodumovOtherM1586: issues with MOLEX?

Urs Langenegger wrote:
Module M1586 had passed the full qualification on 20/02/27. I had had to re-insert the cable in the Molex connector for it to become programmable.

On 2020/03/09, I tried to re-test M1586, but it was not programmable. Visual inspection revealed nothing to me. I did re-insert the cable once again, but this time this did not help.

Maybe one should try again re-inserting the cable.

Maybe these issues are an indication that the module (MOLEX) is flaky.


Most likely the cable contacts caused such behavior since they look damaged.
After changing the cable module do not show any more problems.
  155   Sun Mar 29 18:14:59 2020 danek kotlinskiOtherModules 1544 & 1563 in gelpack
The 2 bad modules:
M1544
M1563
Have been moved to gelpacks.
  175   Thu Apr 2 17:10:08 2020 Andrey StarodumovOtherCap glued to M1618
M1618 has been tested (FT ) on March 23 without a protection cap.
I have no idea how it's happened...
Today cap is glued and module passed Reception test again and it's A.
I think we do not need to repeat FT for this module.
I put it in a tray with good modules.
  210   Tue Apr 14 16:54:31 2020 Andrey StarodumovOtherM1633 cable disconnected
We do not have any more module holders.
M1633 was in "Module doctor" tray.
I took module off the holder and put it in a gel-pak.
  237   Wed Apr 29 08:48:06 2020 Urs LangeneggerOtherM1539
M1539 showed no readout. I tried, all without success,
- reconnecting the cable to the adapter multiple times
- connecting to the adapter in the blue box
- reconnecting the cable to the MOLEX on the module
  258   Mon May 11 14:14:05 2020 danek kotlinskiOtherM1606
On Friday I have tested M1606 at room temperature in the red cold box.
Previously it was reported that trimming does not work for ROC2.

In this test trimming was fine, only 11 pixels failed it.
See the attached 1D and 2D histograms. There is small side peak at about vcal=56 with ~100 pixels.
But this should not be a too big problem?

Also the Pulse height map looks good and the reconstructed pulse height at vcal=70
gives vcal=68.1 with rms=4.2, see the attached plot.

So I conclude that this module is fine.
Attachment 1: m1606_roc2_thr_1d.png
m1606_roc2_thr_1d.png
Attachment 2: m1606_roc2_thr_2d.png
m1606_roc2_thr_2d.png
Attachment 3: m1606_roc2_ph70.png
m1606_roc2_ph70.png
  259   Mon May 11 14:40:16 2020 danek kotlinskiOtherM1582
On Friday I have tested the module M1582 at room temperature in the blue box.
The report in MoreWeb says that this module has problems with trimming 190 pixels in ROC1.

I see not problem in ROC1. The average threshold is 50 with rms=1.37. Only 1 pixel is in the 0 bin.
See the attached 1d and 2d plots.

Also the PH looks good. The vcal 70 PH map is reconstructed at vcal 70.3 with rms of 3.9.
5159 pixels have valid gain calibrations.

I conclude that this module is fine.
Maybe it is again a DTB problem, as reported by Andrey.
D.
Attachment 1: m1582_roc1_thr_1d.png
m1582_roc1_thr_1d.png
Attachment 2: m1582_roc1_thr_2d.png
m1582_roc1_thr_2d.png
Attachment 3: m1582_roc1_ph70.png
m1582_roc1_ph70.png
  264   Wed May 13 17:57:45 2020 Andrey StarodumovOtherL1_DATA backup
L1_DATA files are backed up to the LaCie disk
  275   Tue May 26 23:00:50 2020 Dinko FerencekOtherProblem with external disk filling up too quickly
The external hard disk (LaCie) used to back up the L1 replacement data completely filled up after transferring ~70 GB worth of data even though its capacity is 2 TB. The backup consists of copying all .tar files and the WebOutput/ subfolder from /home/l_tester/L1_DATA/ to /media/l_tester/LaCie/L1_DATA/ The corresponding rsync command is

rsync -avPSh --include="M????_*.tar" --include="WebOutput/***" --exclude="*" /home/l_tester/L1_DATA/ /media/l_tester/LaCie/L1_DATA/

It was discovered that /home/l_tester/L1_DATA/WebOutput/ was by mistake duplicated inside /media/l_tester/LaCie/L1_DATA/WebOutput/ However, this could still not explain the full disk.

The size of the tar files looked fine but /media/l_tester/LaCie/L1_DATA/WebOutput/ was 1.8 TB in size while /home/l_tester/L1_DATA/WebOutput/ was taking up only 50 GB and apart from the above-mentioned duplication, there was no other obvious duplication.

It turned out the file system on the external hard disk had a block size of 512 KB which is unusually large. This is typically set to 4 KB. In practice this meant that every file (and even folder), no matter how small, always occupied at least 512 KB on the disk. For example, I saw the following

l_tester@pc11366:~$ cd /media/l_tester/LaCie/
l_tester@pc11366:/media/l_tester/LaCie$ du -hs Warranty.pdf
512K Warranty.pdf
l_tester@pc11366:/media/l_tester/LaCie$ du -hs --apparent-size Warranty.pdf
94K Warranty.pdf

And in a case like ours, where there are a lot of subfolders and files, many of whom are small, a lot of disk space is effectively wasted.

The file system used on the external disk was exFAT. According to this page, the default block size (in the page they call it the cluster size) for the exFAT file system scales with the drive size and this is the likely reason why the size of 512 KB was used (however, 512 KB is still larger than the largest block size used by default). The main partition on the external disk was finally reformatted as follows

sudo mkfs.exfat -n LaCie -s 8 /dev/sdd2

which set the block size to 4 KB.
  303   Fri Sep 18 15:45:21 2020 Andrey StarodumovOtherM2211 and M2122
I made a mistake and instead M2122 used ID M2211 in the .ini file.
Hence now we do not have entry for M2122 but have 2 entries for M2211: one is of Sept 18 and another one called old of Sep 17).
Test results of Sep 17 are for 2211
Test results of Sep 18 are for 2122

To be corrected later
  307   Mon Jan 18 13:53:44 2021 Andrey StarodumovOtherDTB tests
M2217
--->DTB 154 (one of the red cold box setup):
- flat cable (with HV):
>adctest
clk low = -231.0 mV high= 212.0 mV amplitude = 443.0 mVpp (differential)
ctr low = -246.0 mV high= 235.0 mV amplitude = 481.0 mVpp (differential)
sda low = -250.0 mV high= 223.0 mV amplitude = 473.0 mVpp (differential)
rda low = -65.0 mV high= 48.0 mV amplitude = 113.0 mVpp (differential)
sdata1 low = -171.0 mV high= 155.0 mV amplitude = 326.0 mVpp (differential)
sdata2 low = -173.0 mV high= 149.0 mV amplitude = 322.0 mVpp (differential)

- twisted pairs cable:
several ERRORS during VthrCompCalDel and PixelAlive: ERROR: <datapipe.cc/CheckEventID:L486> Channel 2 Event ID mismatch: local ID (119) != TBM ID (120)
>adctest
clk low = -196.0 mV high= 177.0 mV amplitude = 373.0 mVpp (differential)
ctr low = -245.0 mV high= 236.0 mV amplitude = 481.0 mVpp (differential)
sda low = -249.0 mV high= 224.0 mV amplitude = 473.0 mVpp (differential)
rda low = -65.0 mV high= 55.0 mV amplitude = 120.0 mVpp (differential)
sdata1 low = -120.0 mV high= 101.0 mV amplitude = 221.0 mVpp (differential)
sdata2 low = -124.0 mV high= 102.0 mV amplitude = 226.0 mVpp (differential)

GOOD with flat cable but r/o ERRORs with twisted pairs cable


--->DTB 126 twisted pairs cable:
VthrCompCalDel test: many errors like
ERROR: <datapipe.cc/CheckEventID:L486> Channel 2 Event ID mismatch: local ID (68) != TBM ID (69),
test not converged!
>adctest
clk low = -193.0 mV high= 181.0 mV amplitude = 374.0 mVpp (differential)
ctr low = -252.0 mV high= 242.0 mV amplitude = 494.0 mVpp (differential)
sda low = -243.0 mV high= 227.0 mV amplitude = 470.0 mVpp (differential)
rda low = -67.0 mV high= 47.0 mV amplitude = 114.0 mVpp (differential)
sdata1 low = -122.0 mV high= 103.0 mV amplitude = 225.0 mVpp (differential)
sdata2 low = -120.0 mV high= 107.0 mV amplitude = 227.0 mVpp (differential)

BAD DTB!!!


--->DTB 140
VthrCompCalDel test: many errors like
ERROR:[14:19:38.397] WARNING: Detected DESER400 trailer error bits: "CODE ERROR"
[14:19:38.398] ERROR: <datapipe.cc/CheckEventID:L486> Channel 2 Event ID mismatch: local ID (4) != TBM ID (5)
test not converged!
>adctest
clk low = -194.0 mV high= 184.0 mV amplitude = 378.0 mVpp (differential)
ctr low = -248.0 mV high= 248.0 mV amplitude = 496.0 mVpp (differential)
sda low = -231.0 mV high= 230.0 mV amplitude = 461.0 mVpp (differential)
rda low = -67.0 mV high= 56.0 mV amplitude = 123.0 mVpp (differential)
sdata1 low = -123.0 mV high= 98.0 mV amplitude = 221.0 mVpp (differential)
sdata2 low = -117.0 mV high= 109.0 mV amplitude = 226.0 mVpp (differential)

BAD DTB!!!

--->DTB 172
VthrCompCalDel and PixelAlive tests: many errors like
[14:23:49.222] WARNING: Detected DESER400 trailer error bits: "CODE ERROR"
[14:23:49.222] ERROR: <datapipe.cc/CheckEventID:L486> Channel 2 Event ID mismatch: local ID (192) != TBM ID (193)
[14:24:19.902] WARNING: Detected DESER400 trailer error bits: "CODE ERROR"
[14:24:19.902] ERROR: <datapipe.cc/CheckEventID:L486> Channel 2 Event ID mismatch: local ID (72) != TBM ID (73)
but converged!

DTB WORKS but with ERRORs!!!


--->DTB 139
VthrCompCalDel and PixelAlive tests: many errors like
[14:29:57.160] WARNING: Detected DESER400 trailer error bits: "CODE ERROR"
[14:29:57.160] ERROR: <datapipe.cc/CheckEventID:L486> Channel 2 Event ID mismatch: local ID (159) != TBM ID (160)
and inefficient tornados for ROC8-15
PixelAlive: zero efficiency for all ROCs and many errors

[14:31:25.335] WARNING: Detected DESER400 trailer error bits: "CODE ERROR"
[14:31:25.335] ERROR: <datapipe.cc/CheckEventID:L486> Channel 2 Event ID mismatch: local ID (61) != TBM ID (63)
>adctest
clk low = -191.0 mV high= 194.0 mV amplitude = 385.0 mVpp (differential)
ctr low = -248.0 mV high= 254.0 mV amplitude = 502.0 mVpp (differential)
sda low = -239.0 mV high= 237.0 mV amplitude = 476.0 mVpp (differential)
rda low = -50.0 mV high= 66.0 mV amplitude = 116.0 mVpp (differential)
sdata1 low = -114.0 mV high= 113.0 mV amplitude = 227.0 mVpp (differential)
sdata2 low = -111.0 mV high= 116.0 mV amplitude = 227.0 mVpp (differential)

BAD DTB!!!



--->DTB 94
VthrCompCalDel and PixelAlive tests: many errors like
[14:38:28.017] WARNING: Detected DESER400 trailer error bits: "CODE ERROR"
[14:38:28.017] ERROR: <datapipe.cc/CheckEventID:L486> Channel 2 Event ID mismatch: local ID (133) != TBM ID (134)
and inefficient tornados for all ROC
PixelAlive: zero efficiency for all ROCs and many errors

BAD DTB!!!


--->DTB 65:
[14:44:25.326] ERROR: <PixMonitorFrame.cc/Update:L121> analog current reading unphysical
[14:44:25.326] ERROR: <PixMonitorFrame.cc/Update:L124> digital current reading unphysical

BAD DTB!!!

---> DTB 136 (from my drawer)
[14:47:23.231] ERROR: <hal.cc/status:L112> Testboard not initialized yet!
[14:47:23.231] ERROR: <PixMonitorFrame.cc/Update:L121> analog current reading unphysical
[14:47:23.231] ERROR: <PixMonitorFrame.cc/Update:L124> digital current reading unphysical

BAD DTB!!!


---> DTB 63 (from my drawer)
NO ERRORs!!!
>adctest
clk low = -148.0 mV high= 175.0 mV amplitude = 323.0 mVpp (differential)
ctr low = -245.0 mV high= 230.0 mV amplitude = 475.0 mVpp (differential)
sda low = -251.0 mV high= 225.0 mV amplitude = 476.0 mVpp (differential)
rda low = -61.0 mV high= 60.0 mV amplitude = 121.0 mVpp (differential)
sdata1 low = -121.0 mV high= 101.0 mV amplitude = 222.0 mVpp (differential)
sdata2 low = -124.0 mV high= 102.0 mV amplitude = 226.0 mVpp (differential)

VERY GOOD DTB: no errors with twisted pairs cable!!!


---> DTB 162 (from my drawer)
VthrComp test: many errors like
[14:54:07.161] WARNING: Detected DESER400 trailer error bits: "CODE ERROR"
[14:54:07.161] ERROR: <datapipe.cc/CheckEventID:L486> Channel 2 Event ID mismatch: local ID (195) != TBM ID (196)
[14:54:07.162] ERROR: <datapipe.cc/CheckEventID:L486> Channel 3 Event ID mismatch: local ID (195) != TBM ID (196)
[14:54:07.162] WARNING: Channel 2 ROC 0: Readback start marker after 15 readouts!
and inefficient tornados for ROC8-15

PixelAlive:
many errors like
[14:55:27.941] WARNING: Detected DESER400 trailer error bits: "CODE ERROR"
[14:55:27.941] ERROR: <datapipe.cc/CheckEventID:L486> Channel 2 Event ID mismatch: local ID (193) != TBM ID (194)
[14:55:27.941] ERROR: <datapipe.cc/CheckEventID:L486> Channel 3 Event ID mismatch: local ID (193) != TBM ID (194)
[14:55:27.941] WARNING: Channel 2 ROC 0: Readback start marker after 29 readouts!
and inneficient maps for ROC8-15

>adctest
clk low = -197.0 mV high= 178.0 mV amplitude = 375.0 mVpp (differential)
ctr low = -255.0 mV high= 236.0 mV amplitude = 491.0 mVpp (differential)
sda low = -245.0 mV high= 230.0 mV amplitude = 475.0 mVpp (differential)
rda low = -72.0 mV high= 49.0 mV amplitude = 121.0 mVpp (differential)
sdata1 low = -124.0 mV high= 97.0 mV amplitude = 221.0 mVpp (differential)
sdata2 low = -125.0 mV high= 102.0 mV amplitude = 227.0 mVpp (differential)

WORKS but with ERRORS!!!

---> DTB 170 (from my drawer)
VthrComp test: many errors like
[15:04:20.259] WARNING: Detected DESER400 trailer error bits: "CODE ERROR"
[15:04:20.259] ERROR: <datapipe.cc/CheckEventID:L486> Channel 2 Event ID mismatch: local ID (30) != TBM ID (31)
test mot converged!

BAD DTB!!!


--->DTB 18 (fixed on the red cold box setup):
NO ERRORs!!!
>adctest
clk low = -203.0 mV high= 187.0 mV amplitude = 390.0 mVpp (differential)
ctr low = -252.0 mV high= 236.0 mV amplitude = 488.0 mVpp (differential)
sda low = -248.0 mV high= 230.0 mV amplitude = 478.0 mVpp (differential)
rda low = -60.0 mV high= 48.0 mV amplitude = 108.0 mVpp (differential)
sdata1 low = -114.0 mV high= 111.0 mV amplitude = 225.0 mVpp (differential)
sdata2 low = -120.0 mV high= 109.0 mV amplitude = 229.0 mVpp (differential)

VERY GOOD DTB: no errors with twisted pairs cable!!!
  301   Wed Sep 2 16:09:24 2020 Matej RoguljicModules for PM1595 switch with M1558
Module 1595 was foreseen to go on the inner ladder 5, position -2 (negative two). During the pre-installation test, we saw it had a high leakage current, ~8 microAmps at room temperature. Therefore, we decided to place module 1558 in its place instead of it.
  62   Tue Feb 4 19:35:38 2020 Dinko FerencekModule transfer7 modules prepared for transfer to ETH: 1536, 1537, 1538, 1540, 1541, 1542, 1543
The following modules have been prepared today for transport to ETH for x-ray qualification:
1536, 1537, 1538, 1540, 1541, 1542, 1543

The FullQualification tar files for these modules have been copied to the common CERNBox folder shared with ETH.
  66   Wed Feb 5 16:18:30 2020 Dinko FerencekModule transferModule location tracking added to the assembly spreadsheet
To keep track of the module location, two columns were added to the module assembly spreadsheet https://docs.google.com/spreadsheets/d/12m2ESCLPH5AyWuuYV4KV1c5O_NEy7uP5PQKzVsdr3wQ/edit?usp=sharing that contain the dates when a given module is shipped to ETHZ and when it is returned to PSI.
ELOG V3.1.3-7933898