CLADO: Mixed-Precision Quantization of Deep Neural Networks through Integer Quadratic Programming

Quick start

• Under CLADO_SampleRun We provide a very compact set of CLADO implementations and its end-to-end runs including Ltilde estimation, G Matrix formation, and IQP optimization. HAWQ and MPQCO implementations are also included.
• prep_mpqco_clado.py computes the sensitivities for MPQCO and CLADO; prep_hawq.py computes the Hessian traces, which are used later by optimize.py to compute the sensitivities of HAWQ.
• optimize.py takes the pre-computed sensitivities (traces for HAWQ) and solves the corresponding IQP(for CLADO)/ILP(for HAWQ and MPQCO) problems to get the MPQ decisions. It then evaluates the decisions and report quantized models’ performance.

Other Details and Old Experiments

• Under helpZihao, CLADO_MPQCO_r50.py generates DELTAL_resnet50(MPQCO) and Ltilde_resnet50(CLADO). hawq.py generates HAWQ_DELTAL/TraceEst*.pkl(for HAWQv2/3, only trace estimation).
• Under main folder, variance_study_imagenet_resnet56.py evaluates the decisions generated by CLADO MPQCO, variance_study_imagenet_resnet56_hawq.py evaluates the decisions generated by HAWQ. Evaluated decisions are gathered and stored in evaluated_decisions.pkl.
• After the above steps, use variance_study.ipynb to visualize the results for all methods.

CVXPY MIQP Solver Issue

• Use prob.solve(solver='xxx') to choose the solver of CVXPY problem
• To use 'GUROBI' solver, pip install gurobipy
• To use 'SCIP' solver, conda install -c conda-forge pyscipopt=3.5.0
• IMAGENET: for both KL and noKL MPQ search on A8 x W(2,4,8): 'GUROBI' does the job
• IMAGENET: for noKL MPQ search on A(4,8) x W(4,8): 'GUROBI' non-terminating ; 'SCIP' non-terminating when setting es[es<0]=1e-6 / gives wrong answer when setting es[es<0]=0
• CIFAR100: for noKL MPQ search on A8 x W(2,4,8): 'GUROBI' does the job; 'SCIP' not able to get a solution
• CIFAR100: for KL MPQ search on A8 x W(2,4,8): both 'GUROBI' and 'SCIP' non-terminating (set prob.solve(verbose=True) to see details)

Updates 2022-09-26

• Two more package needed
• conda install -c conda-forge pyscipopt=3.5.0
• pip install cvxpy-base
• Hyperparameter tuning is no longer needed in second phase: optimization
• We formulate the optimization as a Mixed Integer Quadratic Constraint Programming (MIQCP)
• We use the CVXPY solver to solve the MIQCP, PSD approximation of cached_grad was neccessary for MIQCP and was found empirically useful on CIFAR100 sanity check
• Issue of cvxpy MIQCP solver: depends on pyscipopt, may run forever for some unknown reasons on CIFAR100 experiments.
• pyscipopt=3.5.0: no problem for cached_grad = CachedGrad_a248w248c100_resnet56.pkl; stuck for cached_grad = CachedGrad_a248w248c100_resnet56KL.pkl
• pyscipopt=3.1.0: stuck for cached_grad = CachedGrad_a248w248c100_resnet56.pkl

1. CLADO_XXX.ipynb

• binaryWeight: only consider binary quantization (quantize or not) for weights (tested on CIFAR10/100, superiority over naive method confirmed)
• multiWeight: multi-scheme weight quantization (e.g., 2,4,8 bits) for weights (tested on CIFAR10, superiority over naive method confirmed)
• general: multi-scheme (weight,activation) quantization. Problems encountered in sanity check on CIFAR100.

2. How CLADO_XXX.ipynb works

• CLADO_general.ipynb is a more general version of the other two and we should use it
• it perturb pairs of layers and save the change of loss to a pkl file
• the pkl file (e.g.,generala248w248_c10resnet56_calib_kl) contains a 2D matrix capturing the change of loss when perturbing layer (x,y)
• the map of layer index (x,y) to quantization decision of layers are also stored in the pkl file
• the pkl file will be loaded and used to solve the constraint optimization problem
• naive argument, when set to True, cross-layer terms won't be used in optimization
• KL argument, when set to True, perturbation on KL divergence instead of loss will be used as cached gradient in optimization
• so when set KL and naive to be both True, CLADO becomes ZeroQ

3. Sanity check

• Ideally, when set constraint to null, the optimziation should return an 8-bit model, or model that have close-to-FP performance.
• However, we found that the sanity check doesn't pass for CIFAR100-RESNET56 when considering 3x3 options for each layer (A/W 2,4,8 bits)
• The problem is not only for CLADO, but also for ZeroQ
• Negative estimated perturbed loss/KL observed, very likely due to that overly large quantization error is not well captured by first order (ZeroQ) and second order Taylor (CLADO)