BICEP PMAC Logbook

2-Jan-2010, Further Troubleshooting PMAC for BICEP2 at Pole:

New behavior emerged: DK axis DAC output goes nuts for commanded negative values. Took system into open-loop mode and issues commands like "#3o0", "#3o-1" causes runaway condition in positive direction!@!@#!. Based on experience of 26 Dec (below) and given that this is a different PMAC board now, I immediately suspected the ACC28A #2.

Unlike #1, ACC28A #2 is connected with a 16-pin ribbon cable extender which makes it possible to leave connected when sliding out the PMAC breakout on rails. I tried taking out the extender, leaving the cable the same length as ACC28A #1. Problem went away!

TO DO: test 16-way ribbon cable extender to see if it is defective, or if extra length is to blame.

30-Dec-2009, Further Troubleshooting PMAC for BICEP2 at Pole:

Homing on TH axis is not reliable. Discovered that while the capture position m126=$078107, and after much investigation found that AZ axis homing also is also not reliable. Strange oscillations in AZ axis seen at -LIM during home. Suspect firmware may be corrupted? Swapping in: 

PMAC1, TURBO PMAC PCI, ASSY NO. 603588-104 (U3 5F81 p1pci 10/03)
CPU board: barcode C0004VY6, ASSY NO. 603605-105  (TURBO P1 V1.943.0 01-31-07 DELTA TAU) battery 0717
- confirmed all jumpers per 16-Feb-2009 entry below
Pulled old PMAC: 
PMAC1, TURBO PMAC PCI, ASSY NO 603588-103 rev A S/N 863  (U3 5F81 p1pci 10/03)
CPU board: ASSY NO 603605-102 S/N 2237 (TURBO 56311 5C0) (TURBO V1.940 06-11-03 DELTA TAU) battery 0338
Found same symptoms with new PMAC! For example, alternating between "#1j+" and "#1hm" I find that with pmac_code_bicep2.pmc the home trigger misses the AZ reference mark half the time. So old PMAC is fine.

Eventually traced the symptoms to three separate problems with the latest version of the pmac code:

1. AZ oscillations at -LIM homing turnaround were a result of substituting m586 (comm vel) for m581 (actual vel) in plc 11, which dynamically adjusts the AZ gain while tracking. Turns out using m586 results in an instability. I'd made this change only a few days ago--I just changed back.

2. DK zero repeatability: erratic post-trigger movement and large, non-repeatable offsets from the reference mark were the original symptom. Turns out this was probably a result of incorrectly leaving i324=$20800. Bit 11 of this variable turns on high-res capture, appropriate only for ACC-51P encoder capture. The correct setting is bit 11=0, i.e. i324=$20000.

3. In the course of troubleshooting 2, I set m121 and m126 as position-capture registers and then mapped m805 and m806 to read them out so I could plot them in gcp. This confirmed that the PMAC was indeed reliably always seening the reference mark, but often the home move would refuse to trigger on it. This seemed like new pathological behavior on the DK axis, so I compared the AZ axis and then found that it too sometimes missed its home trigger! Turns out the observation was affecting the experiment--reading the position/capture register resets and rearms the capture, which if it happens at the wrong time can defeat the home trigger. I simply took out the temporary debugging code which reads these registers and homing worked great again!

Retuned DK axis. Learned that we can get excellent performance with i334=1, which turns I gain off except when stationary. Nice smooth moves now, and typical tracking error of < 2 arcsec on DK axis, way better than needed.

Defined DK encoder zero: 

encoder_zeros  -322.79, -82.398, 74    # 2009-12-30, approx aligning cryostat zero deg with W (projects to sky to E)
This roughly matches BICEP1 definition of theta=0 pixel orientation on sky (to E at DK=0) to the BICEP2 cryostat theta=0 coordinate axis.

26-Dec-2009, Troubleshooting PMAC for BICEP2 at Pole:

We appear to have a problem with the analog command output of axis 1 (AZ). This was originally appeared today as an instability when moving in the negative direction in AZ under servo control--motion earlier this week on the AZ axis had been fine (after fixing an apparently unrelated problem with the AZ motor encoder, traced to a bad solder job on the motor encoder pins of the motor connector inside the interlock box). We isolated the AZ techron from the mount and connected the 10 Ohm test load. We eventually realized that the command output of the PMAC (DAC1) was non-monotonic, particularly for negative values, e.g. testing in open-loop: 

PMAC:> #1o0

PMAC:> #1o-1

PMAC:> #1o-2

PMAC:> #1o-3
...
gives correct output voltages for some values (0, -1, -3) and wildly incorrect values like +20VDC for others (-2, -7). The pattern appears repeatable. Axis 2 is fine--monotonic output values in open loop mode through the full range of -100 to +100 percent. I did a full reset "$$$***" and repeated. The pattern changes, now some positive output percentages give anomalous DAC1 outputs.

We unplugged the PMAC from the PMAC breakout box to totally isolate it. Using a spare ACC-8 breakout for JMACH1, into which we fed only analog power (AGND(58), A+15V(59), A-15V(60)) and connected the limit switches (+LIM1(51) and -LIM1(53) to AGND(58)) we were able to determine that the same open-loop output commands act properly, i.e. DAC1 output is monotonic and proportional to commanded percentage (around 125mV/percent).

Plugged cables back into the PMAC breakout one by one to determine the source of the problem, and eventually found it was the new flowmeter input, connected to ACC28A #2 ADC3. The problem occurred reproducibly whenever the flowmeter was connected to this input! Furthermore, on the same ACC28A #2 (can't find a serial number, but now labeled "BICEP1 #2, suspect") the input ADC1 doesn't appear to function as an ADC (no response) but DAC1 does respond by a few mV when you plug the flowmeter into it.

Swapped ACC28A #2 out, replacing "BICEP1 #2, suspect" with spare labeled "BICEP2 sn#3". Reordered BNC front-panel ADC inputs and relabeled them AUX0 - AUX4 and FLOW. No issues with corrupted DAC output apparent, but for some reason ADC1 (aux_input[2]) on the new unit doesn't work--it reports values similar (to within a few bits) to ADC4 (aux_input[5]), which is now connected to the flowmeter. [ NOTE 29-Dec-2009, discovered that this last issue was just a typo in pmac code, assigning m805=m308. Corrected and this ACC28A works fine now. ]

16-Feb-2009, Setting up new PMAC for optical test mount:

PMAC1, TURBO PMAC PCI, SN C0004QF4 Jumpers: HRM p 15 - 29 records defaults. Compared every jumper on board XXX main and CPU boards with default settings and notes on changes as recorded in BICEP1 paper copy of PMAC HRM. Made the following changes: 

E85,E87,E88: removed all three jumpers to allow analog +/-15VDC and ground to come in externally.
             According to the HRM this is the default, but the board came with these installed
E90: jumper pins 1-2
E98: moved jumper to pins 2-3 to provide 1.22 MHz DCLK for high accuracy A/D on ACC-28.
E111: jumped pins 2-3 to disable phase, servo, init on J4.  HRM says default is pins 1-2.
E121: removed jumper
E122: removed jumper


ACC-51P, SN C0002TFN
all jumpers confirmed to be default.  Left S1 dipswitches at default: 1-on, 2-off, 3-off, 4-off.
This appears to be for non-Turbo, but let's try it.

15-Jan-2007, AZ tuning at Pole:

Trying to understand the pattern of dependence of thermistor stability (rtc 47) on AZ-scan speed that was documented in http://bicep.southpole.usap.gov/analysis_logbook_north/20070103_stability/ To summarize those results, (1-9, 9 is worst): 

 date   config         ramp1p8  ramp2p0  ramp2p2  ramp2p4  ramp2p6  ramp2p8  ramp3p0  ramp3p2
 11-04  winter 06      3        9        2        3        9        3        4        6
 12-16  p.serv, bad dk 2        9        2        6        3        2        8        3
 12-18  corrected dk   4        8        4        5        4        7        6        4
 12-23  new PID        1        8        1        3        3        3        3        2
 12-24  new PID        1        9        2        3        3        3        6        3

These results are confusing. I observe that at lower scan rates, especially 1.8 and 2.0 d/s, there is a substantial oscillation in AZ velocity pmac.fast_mtr_vel[0]. Could this be the cause of the dependence of thermal instabilities on AZ scan rate? 

Here is the existing tuning:
;s-curve-t  prop.gain   derivative  vel.ff      integral  accel.ff    low pass   fric.ff
i121=1000  i130=2000   i131=15000  i132=9000  i133=30000 i135=140000 i138=-0.85 i168=150 ; 24 aug 2005, smoother az scans

Here is a tuning that smoothes these oscillations:
i121=1000  i130=2000   i131=4000   i132=9000  i133=30000 i135=140000 i138=-0.70 i168=150 ;  2007, smoother az scans
Repeated an AZ stability schedule using this AZ tuning and the new temperature control PID: 
  http://robi/analysis_logbook_south/20070116_az_stability/20070115_az_stability_nopoly.pdf
Not a dramatic difference, 2p0 is still bad and 2p2 still good. Still, if we are going to go with 2p2, I want to kill these oscillations. 5-Feb-2007, Tuning up pmac program at Pole: Replaced the DK coupling today with a stiffer, stainless steel version. Re-aligned DK drive shaft in process (it was off by 0.2 in., not too bad). No explanation for why we saw a 5.0 degree shift in position of DK=0 sometime between Jan 4 and 6. I reset the DK encoder zero in pointing.init: 
#encoder_zeros  -322.79, -82.398, +56.5083  # 2006-12-20 (2007 mark on DK gear & motor)
# note: between Jan 4-6 until Feb 5, DK was somehow shifted by +5.0 degrees (DK=-5.0 to align marks)
encoder_zeros  -322.79, -82.398, -0.930    # 2007-02-05 (after DK coupling replacement, re-aligned 2007 mark on DK gear & motor)

Using bicepPmacterm command 

PMAC:> list prog 1
I verified that the motion program we have been running is pmac_mainprog1_pt.pmc. Combined this and pmac_setup_060627.pmc into pmac_code_bicep_060627.pmc, which reproduces the PMAC state that we ran on BICEP from 06/2006 to 02/2007.

Made these changes, saved in new pmac_code_bicep.pmc, now called v1.01: