 Wed Oct 2 12:41:16 2019, Dinko Ferencek, Module assembly, First production modules assembled 6x
|
First production modules (M1530, M1531, M1532) were built on Monday, Sep. 30 2019.
There was a problem with wire bonding of ROC5 on M1530. There is a small crater on one of the ROC pads which appears to had been created by the BareModule probe card needle.
After initial tests with pXar of the 3 modules, noticed the following:
M1530: data missing from ROCs 4 and 5 (expected based on the wire bond problem on ROC5) but otherwise looks fine
M1531: could set Vana but setvthrcompcaldel and pixelalive not showing any data
M1532: looks fine.
Caps were glued to M1530 and M1532 (damaged cap was glued to M1530) and Reception tests were run before and after gluing. Attached are the bump bonding threshold maps before and after cap gluing and the (before - after) difference for M1530 and M1532. |
|
Thu Sep 26 21:48:56 2019, Dinko Ferencek, Module assembly, Cap gluing training
|
| Today I performed my first cap gluing. As an exercise it was first done on two dummy modules and later on a pre-production module M1522. Before the cap gluing the module M1522 was visually inspected and a Reception test was run. Before the cap gluing everything looked fine. After the cap gluing the module M15222 was again visually inspected and it looked like wire bonds in one of the corners might have been slightly bent and some glue got deposited on some of the wire bonds. The Reception test will be repeated tomorrow. |
 Fri Sep 27 23:22:02 2019, Dinko Ferencek, Module assembly, Cap gluing training
|
| Attached are the bump bonding threshold maps before and after cap gluing and the (before - after) difference for M1522. |
|
Fri Sep 27 22:24:31 2019, Dinko Ferencek, General, Activities on 27. 9. 2019.
|
|
|
Thu Sep 26 22:09:14 2019, Dinko Ferencek, Software, DAC configuration update
|
In accordance with the agreement made in an email thread initiated by Danek, the following changes to DAC settings for ROCs
vsh: 30 -> 8
vclorbias: 30 -> 120
ctrlreg: 0 -> 9
were propagated into existing configuration files in
/home/l_tester/L1_SW/pxar/data/tbm10c/
/home/l_tester/L1_SW/pxar/data/tbm10d/
/home/l_tester/L1_SW/pxar/data/M1522/
on the lab PC at PSI.
It should be noted that ctrlreg was changed to the recommended value for PROC V3. For PROC V4 ctrlreg needs to be set to 17 so this is important to keep in mind when using configuration files and modules built using different versions of PROC. |
|
Thu Sep 26 17:38:07 2019, Matej Roguljic, General, Activities 25.-26.9.
|
| It was planned to test a batch of HDIs these 2 days but the shipment was late. Dinko and Andrei were shown the HDI testing procedure. Matej further worked on the script used to test the HDIs. Dinko glued caps on 2 dummy modules and on M1522. New HDIs arrived on 26th and it looks like the kapton will have to be put underneath them during the testing to prevent sparking. |
|
Thu Sep 26 15:51:30 2019, Dinko Ferencek, Software, pXar code updated
|
pXar code in /home/l_tester/L1_SW/pxar/ on the lab PC was updated yesterday from https://github.com/psi46/pxar/tree/15b956255afb6590931763fd07ed454fb9837fc0 to the latest version https://github.com/psi46/pxar/tree/e17df08c7bbeb8472e8f56ccd2b9d69a113ccdc3 which among other things contains updated DAC settings for ROCs.
All the configs will have to be regenerated before the start of the module qualification. |
|
Thu Sep 12 10:32:27 2019, Matej Roguljic, General, Activities 10.9.-19.9.2019.
|
10.9.
Modules 1504,1505,1520 brought to PSI after irradiation to 1.2 MGy in Zagreb. All 3 were not working when put under test with pXar. 1504 and 1520 couldn't set Vana at all. 1505 could set Vana, but timing couldn't be found and pixels weren't responding. Inspection under the microscope showed some "white-greenish" deposits on the metal pads on the HDIs which are most likely shorting the ROC pads causing the module to be non-responsive
11.9.
Removed cap from 1504 to take a better look under the microscope and some photos. The good news is that the glue holds really well. The deposition looks like a crystal growth. It reminded Danek of the humidity issues they had at CERN in 2014. Touching it with a needle showed that it is not completely solid, like "wet-snow", however, on some pads, the deposition was more solid. There was also small transparent liquid deposits observed. Prepared 3 modules for cap gluing training; 1047, 10X1, 10X2. Only 1047 works (partially) and Reception test was performed on it. Two gluing jigs prepared in the assembly lab.
12.9.
Ran reception on 1509, it will go to ETH for irradiation (not really sure if it's ETH). Further tested modules 10X1 and 10X2. X2 should have hubID set to 17,25 after which the ROCs are programmable, however, there is no readout. X1 should have hudID 31,30; but only TBM with hubID 30 works, ROCs are programmable, but no readout as well. Glued protection cap to a module 10X1. Roughly 4 minutes after the start of mixing, the glue starts to become fairly solid which gives us a constraint on the time it takes us to apply it to a module.
13.9.
Glued caps on 1508 and 1047 using the jig further from the entrance door of the assembly lab. 1508 was glued first and had an asymmetrically placed cap (one side of the cap quite close to the wirebonds). The stage was adjusted so the cap glued to 1047 was properly placed. An unused HDI was found 8-011, it was tested and found to be working until the HV test where it started sparking at -1000V, similar to 8-010. There are burn marks visible under the microscope on the same position as the ones on 8-010.
16.9.
Investigated further the sparking issue with the HDIs. Ed Bartz took a look at the photos and identified that the damaged part of the TBM is the protection diode of pin 46.
17.9.
Equipped the coldbox with 4 DTBs which makes it possible to do qualification on 4 modules at the same time. Investigated phase2 ROC-HDI gluing jig. |
|
Thu Sep 19 14:20:07 2019, Matej Roguljic, General, Activities 10.9.-19.9.2019.
|
| Matej Roguljic wrote: | 10.9.
Modules 1504,1505,1520 brought to PSI after irradiation to 1.2 MGy in Zagreb. All 3 were not working when put under test with pXar. 1504 and 1520 couldn't set Vana at all. 1505 could set Vana, but timing couldn't be found and pixels weren't responding. Inspection under the microscope showed some "white-greenish" deposits on the metal pads on the HDIs which are most likely shorting the ROC pads causing the module to be non-responsive
11.9.
Removed cap from 1504 to take a better look under the microscope and some photos. The good news is that the glue holds really well. The deposition looks like a crystal growth. It reminded Danek of the humidity issues they had at CERN in 2014. Touching it with a needle showed that it is not completely solid, like "wet-snow", however, on some pads, the deposition was more solid. There was also small transparent liquid deposits observed. Prepared 3 modules for cap gluing training; 1047, 10X1, 10X2. Only 1047 works (partially) and Reception test was performed on it. Two gluing jigs prepared in the assembly lab.
12.9.
Ran reception on 1509, it will go to ETH for irradiation (not really sure if it's ETH). Further tested modules 10X1 and 10X2. X2 should have hubID set to 17,25 after which the ROCs are programmable, however, there is no readout. X1 should have hudID 31,30; but only TBM with hubID 30 works, ROCs are programmable, but no readout as well. Glued protection cap to a module 10X1. Roughly 4 minutes after the start of mixing, the glue starts to become fairly solid which gives us a constraint on the time it takes us to apply it to a module.
13.9.
Glued caps on 1508 and 1047 using the jig further from the entrance door of the assembly lab. 1508 was glued first and had an asymmetrically placed cap (one side of the cap quite close to the wirebonds). The stage was adjusted so the cap glued to 1047 was properly placed. An unused HDI was found 8-011, it was tested and found to be working until the HV test where it started sparking at -1000V, similar to 8-010. There are burn marks visible under the microscope on the same position as the ones on 8-010.
16.9.
Investigated further the sparking issue with the HDIs. Ed Bartz took a look at the photos and identified that the damaged part of the TBM is the protection diode of pin 46.
17.9.
Equipped the coldbox with 4 DTBs which makes it possible to do qualification on 4 modules at the same time. Investigated phase2 ROC-HDI gluing jig. |
18.9.
Decided on a final HDI irradiation test. We will have an HDI without components in a bag filled with nitrogen (N2 bag), HDI with components in N2 bag, Dummy module (HDI with cap glued on it) in N2 bag, Dummy module not in an N2 bag and and an HDI with components cleaned by Silvan not in an N2 bag. The idea is to expose them to irradiation in a dry environment to check if there will be weird deposits on the metallic pads of the HDI. With the 2 samples not in an N2 bag we will see if there are some deposits again. To do so, before irradiation we will inspect them under a microscope and take photos and inspect them immediately after irradiation.
Furthermore, a possible explanation for the HDI sparking problem was devised and solutions, if necessary. It is described in more details in a separate log entry. |
|
Tue Sep 10 15:18:20 2019, Matej Roguljic, Other, Modules 1504, 1505, 1520 irradiation report
|
| Modules 1504, 1505 and 1520 were taken to Zagreb for irradiation on 13.08.19. The goal was to check the v4 behavior after irradiation. They were irradiated to 1.2 MGy and returned to PSI on 9th of September. Upon testing them at PSI, they all had issues. ROCs on M1504 and M1520 were not programmable at all, changing Vana had no effect. Vana could be set on M1505 while targeting Iana = 28 mA/ROC, however, no timing could be found for it and no working pixel could be found. Andrey and Matej took the modules under the microscope and saw a "greenish" deposits on HDI metal pads of unknown origin. There was also a bit of liquid near the HDI ID on M1520, but not on others. The residue could be shorting some pads causing issues on the module. It is still unclear whether the residue comes from the HDI, cap or is it introduced during irradiation by something in Zagreb. |
|
Fri Sep 13 15:06:51 2019, Andrey Starodumov, Other, Modules 1504, 1505, 1520 irradiation report
|
It was discovered that these modules were stored in Zagreb in the climatic lab where T was about +22C. These modules have been transported from Co60 irradiation facility to the lab on open air with T>30C and RH>70%. Water in the air under the cap condensated on the surface of HDIs and diluted residuals (from soldering, passivation etc), that after remains liquid or crystallised.
The irradiation itself does not course any damage. This is also confirmed by the fact that after two previous irradiations in Jan and Jul 2019 of modules and HDIs, samples remained in a good shape without any residuals on HSI surfaces. These samples have been kept in an office where T and RH were similar to the outside and not in the climatic lab.
We consider the the case is understood and closed. |
|
Thu Sep 19 00:52:24 2019, Dinko Ferencek, Other, Modules 1504, 1505, 1520 irradiation report
|
| Andrey Starodumov wrote: | It was discovered that these modules were stored in Zagreb in the climatic lab where T was about +22C. These modules have been transported from Co60 irradiation facility to the lab on open air with T>30C and RH>70%. Water in the air under the cap condensated on the surface of HDIs and diluted residuals (from soldering, passivation etc), that after remains liquid or crystallised.
The irradiation itself does not course any damage. This is also confirmed by the fact that after two previous irradiations in Jan and Jul 2019 of modules and HDIs, samples remained in a good shape without any residuals on HSI surfaces. These samples have been kept in an office where T and RH were similar to the outside and not in the climatic lab.
We consider the the case is understood and closed. |
Just to clarify. The modules were not transported from Co60 the irradiation facility to the lab in open air but inside a closed Petri dish. Otherwise, there would be no risk of water condensation if the air surrounding modules was allowed to quickly mix with the air-conditioned lab air. Here the problem arose from the fact that it was not only the modules that were brought inside the lab but they were brought inside a pocket of the outside air. A closed Petri dish is not airtight but it significantly reduces mixing of the air inside the Petri dish with the surrounding lab air making it slower than the rate at which the Petri dish and the module inside it were cooling down once brought inside the lab. This could have led to water condensation if the pocket of air trapped inside the Petri dish was warm and humid and had a dew point above the lab air temperature. To prevent this from happening, the solution should be relatively simple and it would be to open the Petri dish and uncover modules before bringing them inside the lab. That way the exchange of air will be faster and the risk of condensation will be basically gone because a warm module will be quickly surrounded by the lab air which will not condense on a warmer surface.
However, there is an additional twist in this particular case. On Aug. 22, when the modules were brought in the lab, there was a thunderstorm in Zagreb in the early afternoon (https://www.zagreb.info/aktualno/zagreb-je-zahvatila-oluja-munje-i-jaka-kisa-nad-vecim-dijelom-grada/227156) with temperature around 21.5 C and RH around 75% at the time the modules were transported (around 15:20), and the whole day was relatively fresh and humid. The outside air on that day would definitely not lead to water condensation in the lab. However, before being brought in the lab, the modules were sitting in a room in the building where the Co60 irradiation facility is located so the air inside the Petri dish was likely similar to the air inside that room (the modules were sitting there for a while and there was enough time for the air temperature and humidity to equalize) and there was not much time to mix with the outside when being transported from one building to another. Unfortunately, there are no measurements of the air temperature and humidity in that room. However, it is worth mentioning that the previous day, Aug. 21, was not very hot and humid with midday temperature around 26 C and RH around 50%. It is therefore likely that the air inside that room and consequently inside the Petri dish was not very hot and humid, making the hypothesis of water condensation in the lab, if not improbable, certainly less likely.
Either way, more careful handling of modules will be needed. |
|
Wed Sep 18 15:45:45 2019, Matej Roguljic, Module assembly, HDI sparking under HV test 7x
|
HDI 8010 passed all test except the HV. Under the HV test, some sparks could be heard and even seen by eye on the right TBM (closer to the HV pad!). Testing it again, showed that it was indeed damaged by the spark. Sparking craters could be seen with the microscope between pads 4 and 5 of the TBM.
The same thing happened a month later, on HDI 8012. Again, it passed all the test until HV when sparks appeared and damaged the TBM, and, curiously enough, the damage was on the same position on the TBM as with 8010.
HDI 8011 was put under the HV test as well, however, this one didn't have any wires bonded to the TBMs because gold plating was missing on several of the wire bonds. Sparking could be heard, but we couldn't located where the damage happened. There is one potential candidate, shown on the photo attached to this log entry.
On 18.09. a possible explanation for the sparking was thought of. Wolfram and Matej put one of the "faulty" HDIs under HV test and noticed that sometimes a spark could be seen between the HDI handle and the aluminum base plate! It looks like the HV pin rests near the border of the HV pad and the ending of the HDI. Because of that, there is a discharge from the HV pin to the HDI handle. The discharge connects to the ROC wire bond pads closest to the HV pad providing a way to affect the TBM! This is further strengthened by the fact that the damaged TBM pad is connected to the first four ROCs closest to the HV pad. To test this, we put kapton tape on the HDI handle, near the HV pad and tested the HDI again and there were no sparks.
The HDIs to be used for L1 replacement are slightly longer to prevent HV jumping to the sensor. Incidentally, this might also solve the sparking issue. If the issue is not solved on the longer HDIs, HDI holders will have to be further isolated in the region close to the HV pad. |
|
Wed Aug 7 18:01:06 2019, Matej Roguljic, Cold box tests, PhOptimization problem
|
| August 7 - we found out that PhOptimization algorithm starts leaking memory if it fails to find proper values. If running one setup, the test might go through. However, if multiple modules are tested in parallel and they all start leaking memory at the same time, the system will kill one testboard process (or two if necessary), causing the unlucky testboard(s) to get stuck (powered on, HV on, but no tests running) until they are reset. Current solution to this problem is to simply omit the PhOptimization if the fulltest procedure. We left gainPedestal, since it will use default Ph dac values. |
|
Wed Aug 7 12:24:14 2019, Matej Roguljic, Documentation, Activities 31.7.-9.8.2019.
|
31.7.
Matej tested 4 HDIs (8030-8033 and 8010). 8010 had a faulty TBM which was replaced, however, during subsequent tests, other TBM (the one which was working fine!) started sparking during HV test. Burn marks visible under the microscope in the top-right corner of the right TBM, between wire-bond pads. Other HDIs were fine and were glued to 2 v4 modules (1523 and 1529) and 1 v3 module (1526).
1.8.
Prepared the software and hardware for the full-qualification of multiple modules in parallel. Ran reception test on M1520 and M1522.
2.8.
Ran full qualification on M1520 and M1522. It consisted of qualification at -20, IV@-20, 5 cycles between -20 and +10, qualification at -20, IV@-20, qualification at +10 and IV@+10. This set of tests will from now on simply be called "full qualification". For some reason, second qualification at -20 failed. M1522 shows significantly high leakage current compared to previous tests
5.8.
Boxes 2 and 9 have been filled with L1 module trays. Each tray can hold 4 modules and each box contains 13 trays. Box 9 filled with humidity collecting balls. M1520 and M1522 placed in the box 9 which is the box that should be filled first. Reception test was run on M1523 and M1526. There were a lot of issues with M1523 in the address decoding test affecting other tests as well. What was found out is that the third dac, Vsh, was set to the wrong value by default if one was making module configuration folder using mkConfig script. It was 30 and should be 8. This should be propagated in pXar later on.
6.8.
Started running reception tests with 3 modules (1523,1526,1529) in parallel, but with the corrected value of Vsh (8). Several issues were observed.
First, one of the DTBs had a connection timeout which appeared every time we tried the reception test. Changing the DTB didn't work. This was solved by plugging the USB cable on the PC side to a different USB slot
Second, pretest in the reception test did timing test which we deemed unnecessary so it was removed.
Third, results for 1523 and 1529 didn't look correct. It was as if there was a problem with either timing or dac values. Comparing the logfiles of the test and the pxar files, it was found out that the timing settings were correct. The issue was in the dac settings taken by elComandante. What we forgot was that elComandante does NOT take dac settings frmo the module folder, but rather from generic tbm folders like tbm10d. This is set in the elComandante.config. The issue was that tbm10d folder had wrong values for Vsh (30, instead of 8) and ctrlreg (0 instead of 17). After this correction reception tests ran fine.
PixPhOptimization.cc was edited to correct the starting values of the parameters for the PhOptimization test. The old values were sometimes causing the tests to algorithm to fail even though the chip was fine.
Started full qualifications of all 3 modules, but there were some issues with the coldbox. It was noticed when it was not cooling at the start of full characterization. The warning on the coldbox interface read "Geber-stillstand". Danek managed to make it go away by stopping the program, even though no program was being run.
Finally, full qualification started around 15:00, finished at 22:40.
M1523 failed first qualification at -20, but finished other two qualifications. IV curves for that module were taken while M1529 was stuck and connected to HV as well meaning that the readings are actually sum of IVs from these two modules.
M1529 failed first qualification at -20, few seconds apart from M1523 failure. One of them failed during the Gain Pedestal, other one on PhOptimization. It also failed other two qualifications (PhOptimization and GainPedestal)
M1526 had no issues in qualification, however, it was graded C. One chip was problematic in most of the tests, while 4 others had issues with TrimBits. This is a bit suspicious so we'll investigate it further.
7.8.
Investigating problems with qualifications on testboards 0 and 1. We noticed that it happens during PhOptimization or GainPedestal. Also investigating why M1526 failed trimbits test on several ROCs. The cause was identified as a memory leak (several Gbs!) during a PhOptimization scan. Separate log made for this.
8.8.
Started full qualification WITHOUT PhOptimization at 9:15 on modules 1523, 1526, 1529. Finished after 8 hours. 1529 had a reception test error during fulltest@+10, other tests ran fine. Glued protection caps on 4 modules - 1504,1505,1509,1520 using 2 component epoxy adhesive.
9.8.
Running fulltest@-20 and +10 for modules 1504, 1505, 1520. The plan is to irradiate them in Zagreb and evaluate the changes induced by radiation. |
|
Wed Aug 7 11:49:07 2019, Matej Roguljic, Module assembly, HDI glue irradiation tests 10x
|
Six different glues were used to glue the cap on six dummy modules. They were all irradiated to 120 MRad in Zagreb after which they were taken to PSI and tested with the tweezers under the microscope.
Standard glue (Dow-cornig), used in CMS - it became "two-component". Solid part on the capacitors and the liquid part on the cap. Surface tension was actually holding the cap quite strongly. Silvan deemed it the second best glue
Two component epoxy adhesive - the strongest glue out of all tested. The only drawback of it was that during the application of the glue to the capacitors, it left puddles around some of them since it is quite liquid before curing. This might have been prevented if we had a proper glue stamp. The glue stamp used at the time contained too much excess glue, and if another one was printed, with small grooves where the capacitors are, the amount of glue transferred should be lower and no puddles should appear. This glue was graded the best of all the tested glues. EDIT: the glue stamp with smaller grooves was printed and now there are no puddles anymore.
SG-20 (black) - sticks, but not really well. It was also quite soft and malleable. Because it doesn't stick very well, it is not recommended for gluing the cap.
WS-200 - almost identical to the SG-20, just a different color.
Terosan MS939 - doesn't stick, not recommended.
Ergo 6521 - doesn't stick, not recommended.
Took photos of them under the microscope. There are two photos for each glue. First one is while the cap is at rest, while on the second there is an upwards force applied with the tweezers. All of them except the two component epoxy adhesive detached when force was applied. With the epoxy adehsive, the module started lifting, but the glue held. |
|
Tue Aug 6 16:06:45 2019, Matej Roguljic, Other, Vsh and ctrlreg for v4 chips
|
| The recommended default value for Vsh is 8, but the current version of pXar has it as 30. One should remember this when making new configuration folders from mkConfig. The recommended value of ctrlreg is 17. |
|
Tue Aug 6 16:01:17 2019, Matej Roguljic, Software, Wrong dac settings - elComandante
|
| If it seems that elComandante is taking wrong dac settings for tests like Reception test or full qualification, one should remember that it does NOT read values from module specific folders like "M1523", but rather from tbm-specific folders like "tbm10d". The folders from which the dacs are taken are listed in "elComandante.config", the lines which look like "tbm10d:tbm10d". |
|
Tue Aug 6 15:15:00 2019, Matej Roguljic, General, Activities 19.6.-4.7.2019.
|
17.6.-21.6.
Andrei and Matej were at PSI working on the cap gluing setup
Andrei cap glued M2029 (Box9,T110)
Matej cap glued M2331
M2331 was tested before and after cap gluing
M2233 was taken to Zagreb for irradiatio, cable of the mdule is in Box4, T040
1.7.
HDIs 3019 and 3020 were tested and both failed, results in the HDIresults
Performed characterization on one v4 chip, it will be irradiated in Zagreb and restested
2.7.
Silvan bonded the M1520 so Andrei and Matej performed full qualification on it (-20 and +10) and it took 5 hours
3.7.
Glued caps on 3 dummy modules with 3 different glues.
HDI 1317-3-008 with two component epoxy glue
HDI 1317-3-010 with SG-20 silicon glue
HDI 1317-3-004 with WS-200 silicon glue
4.7.
3 dummy modules taken to Zagreb for irradiation by Matej
M1508 was also taken |
|
Tue Aug 6 14:30:00 2019, Dinko Ferencek, Documentation, Jumo Imago 500 manuals
|
| Manuals can be found at https://www.manualslib.com/products/Jumo-Imago-500-8786441.html |
|