Introduction

If you're reading this, and you're not me, than you are a sorry bastard, and you probably deserve whatever's coming to you.

Other Interesting Pages

My Guide to COMPHEP

FAQ

General

From My Experiences, It Seems That the CMSSW Programmers are Working for ATLAS. Is This True?

Never ascribe to malice what can be satisfactorally explained by pure incompetence.

Can They Really Be This Incompetent?

It pushes the boundaries of disbelief, but it is still statistically possible for them to be this incompetent.

Are There General Instructions to Run CMSSW?

Yes.

Do You Know Where They Are?

Of course not. Who do you think I am? God?

Coding

How Should I Name My Variables?

I come from the school of large variable names, preferring to use names like Lepton1Px? to describe the x momentum of the first lepton. I also prefer to declare all my variables at the start of the program. Let me explain this, there is no shortage of space on our disk servers that can be alleviated by abbreviating FakePhotonMomentum? to FPM. It doesn't make you look professional. It only makes you unreadable.

But all my Variables are Named Things Like HXK_477, and are Defined Halfway Through the Code. What Should I Do?

If you're smart, you should commit suicide now before I can catch up to you.

But I Only Found This Page Through Google. Why do You Care What I Code?

Look, the kind of people who use unclear variable names, and define them in secret, make life a living hell for physics students. All physics students are sworn to hunt down anyone who is stupid enough to keep doing this, before their infection can spread.

My Professor Reuses Variables when he Codes? Should I?

Your professor is a relic of the age when the memory of the average physicist was so inefficient that they were unable to remember the variables they used if the number of variables exceeded 100. In order to prevent physicists from overloading their internal memory and suffering a mental breakdown, some professors reused variables to do multiple things. This is also because physicists of this variety were attempting to prevent anyone from understanding their research. We no longer have to do this. If you do it, we will trash your code and write something useful for a change.

Instructions

Basics

scram

SCRAM, or scram, or scramv1 (the actual command as of October 4th, 2007) is the general build command that creates all executables. It also creates project directories, checks for distributions, and does everything else too. Almost everything that you do that does not involve running cmsRun will involve scram. You can use scram to:

• Check the available CMSSW releases:

     scramv1 list CMSSW
     

• Check out a new project directory for CMSSW

     scramv1 project CMSSW [CMSSW Release]
     

• Build the directory you are in

     scramv1 -b
     

cmsRun

Unlike some other project software, CMSSW is built very centrally. There is only one executable imaginable, cmsRun. In the course of your analysis you create modules, Producers and Analyzers and other bits and pieces, and build them into self-contained components using scram. Then you write .cfg, .cff, and .cfi files that describe the attributes of these modules and pass variables onto your code. However, everything that you run will be run using cmsRun as an executable. Therefore, all the actual work you do will look like this:

cmsRun someplace/something.cfg

This makes CMSSW very modular and very adaptable. Keep track of what you're doing though. It can easily get away from you.

CRAB

The CMS Remote Analysis Builder, the software which, in theory, allows you to run analysis code and remote OSG and LCG sites around the word. Not to be confused with crap, an adjective, which is often used to describe the relative utility of CRAB.

Ntuples

The simplest example of an Ntuple is essentially a two dimensional array, designed to hold the data from a certain number of events. Let's create a basic ntuple that is meant to hold the four-momentum of a particle from each of ten different events. You could create an array of size a[4][10], where each element in the [10] represents an event, and each in the [4] a component of four-momentum. So, if you wanted to plot the Px for all the events, you would loop through a[0][i] for all i.

The default root Ntuple structure is called a tree, initialized in a class called TTree. A tree is filled with branches (TBranch), and each branch can be filled with leaves (TLeaf). The tree is essentially a giant, multi-dimensional array, whose number of dimensions can be expanded fairly easily. The tree indexes its information by entries. When you call an entry:

tree->GetEntry(10)

The tree then sets the output of any branch, any leaf within the tree to the value in index 10. Since usually we use this for Monte Carlo, this would allow you to retrieve all the information for the 10th event.

The Ntuple structure allows you to do actual physics-related things to your data. For instance, if you are storing information about leptons and jets in your Ntuple, you can, for each event, examine the kinematic properties of the leptons, and then if they pass a certain criteria, plot the jet distribution. This, as you can imagine, is a powerful physics tool.

Almost everything is stored in Ntuples. The SnT group has its own Ntuple, which is discussed below.

Getting Started

How to Install CMSSW

CMSSW works on the release system. This means that every time you want to use CMSSW, you have to choose a release. During actual CMS operation, the releases will be fairly stable, and you will be using one release for the entirety of your analysis. For the course of time during which CMSSW is still evolving, you are plumb out of luck, and CMSSW versions have a shorter lifetime than a celebrity marriage. This problem is compounded by the fact that many new releases are not fully tested when released - so you get caught in a Catch-22. If you don't stay with the current version you are at the mercy of all manner of known bugs, but if you do go on to the head release, you will be at the mercy of all manner of unknown bugs. Well, you pays your money, you takes your chances.

First, if you're at UCSD, log into the uaf machines (uaf-2 is the only one that runs new distributions as of October 4th, 2007). Once there, you set up for doing CMSSW by the following:

source ~norman/.bash_profile

If you are not on the uaf machines (or I have long since ceased to do work and my directory is erased), the current commands (as of October 4th, 2007) that are relevant are:

source /code/osgcode/cmssoft/cms/setup/bashrc
export SCRAM_ARCH=slc4_ia32_gcc345

This sets up the infrastructure so that you can actually create yourself a release.

Once you have done the initial setup, you need to choose where to create your release. I have a directory within my home directory that I put all my releases in to avoid cluttering up home. You may choose to do this entirely in home. I will refer to whatever directory you choose to put your releases in as your [CMS Directory] and the directory of your actual release as your [Release Directory]. I will also refer to the version you choose to use as your [CMSSW Version].

First you need to create your [Release Directory]. This is done automatically.

cd [CMS Directory]
scramv1 project CMSSW [CMSSW Version]

This introduces the scram command, which is described above in the Basics section. This tells scramv1 to build a project ( a release directory) from the CMSSW tree, using the release that you specify. This release tag is the full release tag. For instance, to build a CMSSW_1_5_4 release, you do:

scramv1 project CMSSW CMSSW_1_5_4

This will result in a flurry of output, looking like:

Creating a developer area based on project CMSSW, version CMSSW_1_5_4
Getting project release area....
Checking SCRAM version....

...

Installation procedure complete.
Developer area located at:

                /home/users/norman/Test/CMSSW_1_5_4

You should now have a directory inside [CMS Directory] called [CMSSW_Version]. It should already have several directories and files in it.

[1030] norman@uaf-2 ~/Test$ ls CMSSW_1_5_4/
bin  config  doc  include  lib  logs  module  python  share  src  test

Our next step will be to actually set us up to run CMSSW. To do this, we have to set up several runtime variables into our environment. The way we do this is:

cd [CMSSW_Version]/src
eval `scramv1 runtime -sh`

replacing -sh with -csh if you are running C-shell for some strange reason. This should only take a second, and will do various invisible, hardly understood things in the background, and eventually you will magically be able to run your software.

How to Install the SnT Framework

All right, now you have a release in some strange version, and you think you're ready to do actual physics in the Surf'n'Turf framework. Except that you don't have the framework, because it's not part of the bare bones standard release. You have to go and get it for yourself. The code is stored in the central CMS CVS repository, at CERN, and you are going to need to first bring it to your directory. So, first go into your source directory.

cd [CMS Directory]/[CMSSW Version]/src

Now you have to take care of CVS. All this magic stuff you've done with environment variables and setup scripts should have defined an alias called cmscvsroot. Run this. This should set up your CVSROOT, the repository you get CVS from.

[1049] norman@uaf-2 ~/Test$ cmscvsroot
[1053] norman@uaf-2 ~/Test$ echo $CVSROOT
:pserver:anonymous@cmscvs.cern.ch:/cvs_server/repositories/

NOTE: As of October 4th, 2007, cmscvsroot does not setup the correct $CVSROOT. You can correct this by doing:

export CVSROOT=:pserver:anonymous@cmscvs.cern.ch:/cvs_server/repositories/CMSSW

Then you need to login.

[1057] norman@uaf-2 ~/Test$ cvs login
Logging in to :pserver:anonymous@cmscvs.cern.ch:2401/cvs_server/repositories/CMSSW
CVS password:

As of this writing, the cvs password is 98passwd. It is highly secure, as you can tell.

After this, you should be able to grab the directories you want.

cvs co -d CMS1 UserCode/Gutsche/CMS1

WARNING: The following was built for CMSSW_1_5_4. It is not expected to work on other releases.

This will take all the files in Oliver Gutsche's user code into a directory called CMS1 in your source directory. Once this is done, you need to arrange and build the release. You can do this simply by running the default installation script, or, as Matt L. suggests, you can alter the script. The default installation script is a perl script, and it builds the entire directory. Matt L. prefers not to build the directory, but if you are a new user, I recommend just running it as described.

norman@uaf-2 ~/Test/CMSSW_1_5_4/src$ cd CMS1/
norman@uaf-2 ~/Test/CMSSW_1_5_4/src/CMS1$ ls Configuration/
CVS  installCMS1.pl  required_packages.txt
norman@uaf-2 ~/Test/CMSSW_1_5_4/src/CMS1$ ./Configuration/installCMS1.pl CMSSW_1_5_4

This should install all the new packages, and then build the entire directory.

NOTE: I have had to reestablish the SCRAM Architecture before doing this step, even though it was set previously.

norman@uaf-2 ~/Test/CMSSW_1_5_4/src/CMS1$ echo $SCRAM_ARCH
slc3_ia32_gcc323
export SCRAM_ARCH=slc4_ia32_gcc345

Operations

CMSSW

CRAB

The basic rundown of a CRAB job is in the crab.cfg, a file that exists in each job.

To run CRAB, do the following

Go into your CRAB working directory. In here you should have a directory containing the CRAB working release, inside of which there should be a file like crab.sh:

• Source crab.sh
• In your working directory, create a directory for your new submission
• In that directory, create a crab.cfg file. A typical CRAB configuration file is explained in the next section.
• run crab -create to create your jobs.
• run crab -submit to submit your jobs.
• Once your jobs have completed (check monitoring), you can retrieve your output by running crab -status and crab -getoutput in that directory

Where the Hell Do I Send My Jobs?

A pressing question for most CRAB newbies. The current CMS production paradigm is to spam every possible site that might run your jobs, in the knowledge that sooner or later your shoddy job handling will result in so many jobs gumming up the works that you will destroy the entire grid, meaning that, even if you get nothing done, nobody else is getting anything done either.

We could be more useful, but why bother?

Unless you're one of those people who thinks that they should play nice and should not deliberately sabotage everyone else in the experiment. In that case, you clearly have no future in particle physics, but you can get some mileage out of deliberate submission.

First you need a dataset. The canonical way of finding a dataset is to go to the:

DBS Discovery Page

Where all the CMS datasets are listed in a pulldown menu under "Primary dataset/MC generators". Note that this includes every dataset in CMS, under every software version it was run off of. This includes all current datasets, and all the older datasets, and all the test datasets, and all the MC datasets for weird exotic effects that were only produced because it made sense for a short term analysis, and it was late, and we were drunk, and that's why we probably have qg->H+X+Y MC somewhere in the background, as well as fifty sets of Z->ee MC that looks all the same. And probably even some stuff from LEP included in there for good measure.

Needless to say, if you don't know what dataset you want, you can spend a lot of time browsing the windows, looking for what you want.

Some things to check while looking for a dataset:

• Do you know what it is (or is the title confusing as hell?)
• Does it have a software release similar to yours?
  • Usually you run on a software release that is either yours, or up to one full level below yours.
• For the software release that you want, how many files does it have? How many events?
• Is the site that this is on a reliable site?

You will, once you decide upon your dataset, need the following pieces of information.

• The dataset name (example: /RelValZEE/CMSSW_1_6_8-RelVal-1196896807/GEN-SIM-DIGI-RECO)
• The site name
• The Storage Element name (example: srm.cern.ch)

According to the way things are specified in the CMS Computing Paradigm, data should only be stored at a relatively small number of sites, the Tier 1s, and Tier 2s. This should amount to under ten Tier 1s, and about fifty Tier 2s worldwide. At the time of this writing, the number of functional Tier 1s is two (CERN and FNAL), and the number of functional Tier 2s is basically limited to the number of USCMS Tier 2s (Currently 7 + FNAL, which is special).

Note: Fermilab is very, very special. It has a monstrous Tier 1 facility, both for storage and analysis, which was explained to me as approximately the equivalent of two different Tier 1s in any other part of the world. To support that, it also has the massive CMS LPC facility, a computing facility that is equivalent to several Tier 2s in terms of computing analysis power. Fermilab is the best equipped, and one of the best run, sites in the world, and has massive amounts of computing power. The downside to this is that everyone from the US will be running there.

Basic details about prominent US sites is below:

Config Files

Ntuples

SnT Ntuple Building

Ntuples can be created in the Surf'n'Turf framework by following a fairly simple procedure - unless you want to do something complicated. Let's start with the simplest method. The Ntupling software is contained in the NtupleMaker directory of the SnT framework (not like that's a big surprise). Included in the test directory should be a file called analyze.cfg that contains everything that you need to build your own ntuple.

In order to run a basic ntuple, you simply want to edit the analyze.cfg file.

First, comment out everything in the:

source = PoolSource
  {
    untracked vstring fileNames = 

 ....

  }

You should also edit the:

replace ntupleMaker.fileName = "/tmp/ntuple.root"

piece to point to where you want your output ntuple to point.

Then, into it add all the files you want to look at. For instance, for the Wgamma ntuples, my analyze.cfg file looks like:

# -*-sh-*-
process A = {

  untracked PSet maxEvents = {untracked int32 input = 1000000}
  untracked PSet options =
  {
    untracked vstring Rethrow = {"ProductNotFound"}
  }
        
  source = PoolSource
  {
    untracked vstring fileNames = 
    {
    # "file:/data/dmytro/samples/RelVal-RelVal151TTbar-1183231184-0000-0CC1FC7F-A127-DC11-88B4-001617C3B6D2.root"
    # "file:/data/dmytro/samples/RelVal-RelVal151Higgs-ZZ-4Mu-1183228327-0000-860585E7-5227-DC11-8227-000423D662EE.root"
    # "dcache:/pnfs/cms/WAX/8/cms1/DiLepton/samples/CMSSW_1_3_1_test/HWW/outputfile_0.root"
    # "/store/RelVal/2007/7/10/RelVal-RelVal152TTbar-1184059625/0000/08BD2009-042F-DC11-A1CE-001617C3B6F0.root"
    # "/store/mc/2007/7/31/CSA07-DrellYan_mumu_40-2099/0001/06A7F6EB-F140-DC11-90D8-00E081439BBA.root"
    "dcap://dcopy-1.local:22138/pnfs/sdsc.edu/data3/cms/phedex/hpi/DiLepton_CMSSW153/Wgamma_cmssw144/reco_0.root",
    "dcap://dcopy-1.local:22138/pnfs/sdsc.edu/data3/cms/phedex/hpi/DiLepton_CMSSW153/Wgamma_cmssw144/reco_10.root",
    "dcap://dcopy-1.local:22138/pnfs/sdsc.edu/data3/cms/phedex/hpi/DiLepton_CMSSW153/Wgamma_cmssw144/reco_11.root",
    "dcap://dcopy-1.local:22138/pnfs/sdsc.edu/data3/cms/phedex/hpi/DiLepton_CMSSW153/Wgamma_cmssw144/reco_12.root",
    "dcap://dcopy-1.local:22138/pnfs/sdsc.edu/data3/cms/phedex/hpi/DiLepton_CMSSW153/Wgamma_cmssw144/reco_13.root",
    "dcap://dcopy-1.local:22138/pnfs/sdsc.edu/data3/cms/phedex/hpi/DiLepton_CMSSW153/Wgamma_cmssw144/reco_14.root",
    "dcap://dcopy-1.local:22138/pnfs/sdsc.edu/data3/cms/phedex/hpi/DiLepton_CMSSW153/Wgamma_cmssw144/reco_18.root",
    "dcap://dcopy-1.local:22138/pnfs/sdsc.edu/data3/cms/phedex/hpi/DiLepton_CMSSW153/Wgamma_cmssw144/reco_2.root",
    "dcap://dcopy-1.local:22138/pnfs/sdsc.edu/data3/cms/phedex/hpi/DiLepton_CMSSW153/Wgamma_cmssw144/reco_3.root",
    "dcap://dcopy-1.local:22138/pnfs/sdsc.edu/data3/cms/phedex/hpi/DiLepton_CMSSW153/Wgamma_cmssw144/reco_4.root",
    "dcap://dcopy-1.local:22138/pnfs/sdsc.edu/data3/cms/phedex/hpi/DiLepton_CMSSW153/Wgamma_cmssw144/reco_5.root",
    "dcap://dcopy-1.local:22138/pnfs/sdsc.edu/data3/cms/phedex/hpi/DiLepton_CMSSW153/Wgamma_cmssw144/reco_6.root",
    "dcap://dcopy-1.local:22138/pnfs/sdsc.edu/data3/cms/phedex/hpi/DiLepton_CMSSW153/Wgamma_cmssw144/reco_7.root",
    "dcap://dcopy-1.local:22138/pnfs/sdsc.edu/data3/cms/phedex/hpi/DiLepton_CMSSW153/Wgamma_cmssw144/reco_8.root",
    "dcap://dcopy-1.local:22138/pnfs/sdsc.edu/data3/cms/phedex/hpi/DiLepton_CMSSW153/Wgamma_cmssw144/reco_9.root"
    }
    # untracked uint32 skipEvents = 0
  }
  
  include "CMS1/NtupleMaker/data/ntupleMaker.cff"
  replace ntupleMaker.fileName = "/home/users/norman/CMS/CMSSW_1_5_4/src/CMS1/ntuple.root"
  # replace ntupleMaker.tightMuonType = "TightTrackerMuons"
  # replace ntupleMaker.looseMuonType = "LooseTrackerMuons"
  # replace ntupleMaker.referenceMuonType = "AllTrackerMuons"
  
  path p = { NtupleMakerSequence }
}

You run this the same way you run everything else in CMSSW:

cmsRun NtupleMaker/test/analyze.cfg

This will create a lot of output, most of it, for incomprehensible reasons, in Standard Error, so you may want to redirect standard output and standard error to a single file.

Ntuple Structure

The SnTuple? is structured in the following manner.

• The master index is the event. Loading the SnTuple? as a tree and running GetEvent? gets one single event.
• In each event there are several collections, associated only by name of the variable (they are not separate branches of the tree):
  • evt holds event information.
  • hyp holds information about our selected objects (tight lepton, loose lepton, and jets(?) )
  • mus holds information about all muons in the event
  • els holds information about all electrons in the event
  • trks holds information about all tracks in the event
  • genjs holds information about generator jets.
  • genps holds information about generator particles.
  • etc.
• There are two types of sub-collections, flat and vectors.
  • Flat includes things that can be described by a single entry per event. For instance, the d0 of the tight lepton is flat. There is only one tight lepton per event, and it has only one d0 value, which is stored in "hyp_lt_d0". Similarly, for this purpose, the four momentum of the tight lepton is considered to be flat. It is stored as a LorentzVector in "hyp_lt_p4". Each time you use tree->GetEntry(i) you call one value of these objects into the branch.
  • Vectors are for items where the number of variables would not be known entering the event. For instance the els collection contains all the electrons in the event (that pass basic selection). Because there is no way of knowing a priori how many electrons you will have, the information is stored in vectors. For instance, if you wanted to know the momentum of every electron in the event, you would call upon the branch "els_p4", which would return you a vector containing one four vector for every electron in the branch.
• You can often relate information between sub-collections. This is useful, because there is information in one collection that may not be present in another.
  • For example, let us say that your leading lepton is an electron, and you want to find out the Sigma Eta Eta of this electron. This is not stored in the hyp collection. Instead, the Sigma Eta Eta for all electrons, including the leading lepton, is stored in the els collection. The branch "els_sigmaEtaEta" returns a vector of all Sigma Eta Eta values for all electrons in the collection in the form of a vector. In order to figure out which value belongs to the electron you're interested in, call "hyp_lt_index", which returns the index position of your electron in your vector. Then use either the vector[] operator, or the vector->at() function to call the precise value you want.

Ntuple Variables

The current (October 5th, 2007) SnT Ntuple is semi-flat, containing only floats, doubles, Lorentz Vectors, and vectors of Lorentz Vectors. In order to find a brief description of each branch, the easiest thing to do is to load the tree into root on the command line, and then run tree->Print(). Each branch is printed with a brief description.

As of this time, the variables are defined separate from the main NtupleMaker? code, spread throughout the streamers. You can see a complete list when the plots are made in NtupleMacros? /pass3/. Variables are defined in CMS1.h, and are filled in myLoopingFunction.C.

Here are some of the more useful features:

Ntuple Known Difficulties

• trks_p4 is not the collection of track momenta, but appears to be an ambiguously determined quantity, and is mostly empty.

Troubleshooting

SCRAM level

scramv1 list fails:

scramv1 list CMSSW
Cannot find Environment file. Area Corrupted? (/code/osgcode/cmssoft/cms/slc4_ia32_gcc345/cms/cmssw/CMSSW_1_6_4)
 Permission denied

SnT Installation fails

[1117] norman@uaf-2 ~/Test/CMSSW_1_6_1/src/CMS1$ ./Configuration/installCMS1.pl CMSSW_1_6_1
Found existing CMS1 installation. Use it and ignore requested tag. Please update by hand if it's necessary
Compiling and building code ...
Cleaning local working directory:module/slc4_ia32_gcc345
Cleaning local working directory:lib/slc4_ia32_gcc345
Cleaning local working directory:tmp/slc4_ia32_gcc345/src
Resetting project timestamp cache:
Cleaning ProductStore directories:
>> Building CMSSW version CMSSW_1_6_1 ----
Entering Package CMS1/Base
Entering library rule at CMS1/Base
>> Compiling /home/users/norman/Test/CMSSW_1_6_1/src/CMS1/Base/src/BaseStreamer.cc
>> Compiling /home/users/norman/Test/CMSSW_1_6_1/src/CMS1/Base/src/Cuts.cc
>> Compiling /home/users/norman/Test/CMSSW_1_6_1/src/CMS1/Base/src/Dump.cc
>> Compiling /home/users/norman/Test/CMSSW_1_6_1/src/CMS1/Base/src/ElectronStreamer.cc
>> Compiling /home/users/norman/Test/CMSSW_1_6_1/src/CMS1/Base/src/ElectronUtil.cc
>> Compiling /home/users/norman/Test/CMSSW_1_6_1/src/CMS1/Base/src/EventData.cc
>> Compiling /home/users/norman/Test/CMSSW_1_6_1/src/CMS1/Base/src/GenJetStreamer.cc
>> Compiling /home/users/norman/Test/CMSSW_1_6_1/src/CMS1/Base/src/GenParticleStreamer.cc
>> Compiling /home/users/norman/Test/CMSSW_1_6_1/src/CMS1/Base/src/JetStreamer.cc
>> Compiling /home/users/norman/Test/CMSSW_1_6_1/src/CMS1/Base/src/MuonStreamer.cc
/home/users/norman/Test/CMSSW_1_6_1/src/CMS1/Base/src/MuonStreamer.cc: In member function `void cms1::MuonStreamer::fill(const reco::Candidate*, bool)':
/home/users/norman/Test/CMSSW_1_6_1/src/CMS1/Base/src/MuonStreamer.cc:82: error: `TM2DCompatibilityLoose' is not a member of `muonid'
/home/users/norman/Test/CMSSW_1_6_1/src/CMS1/Base/src/MuonStreamer.cc:83: error: `TM2DCompatibilityTight' is not a member of `muonid'
gmake: *** [tmp/slc4_ia32_gcc345/src/CMS1/Base/src/CMS1Base/MuonStreamer.o] Error 1

This means that you are trying to use the installation script on a version that is not CMSSW_1_5_4. The installation script does not currently work on other versions of CMSSW, and is not expected to anytime in the near future.

-- MatthewNorman - 03 Oct 2007