My Professional Network, Graphically

A few months ago, I was thinking about my professional network and wondered what it would look like if it were mapped out. It turns out, LinkedIn has a lab project to do just that. I loaded up my profile and a few minutes later, this came out (click for larger version):

Each dot in the above chart represents a person, and each line represents a connection between people. The larger the dot, the closer the relationship. I am at the center, and unique clusters of people become apparent by their interrelationships, and they are grouped together in space.

The lab does not label the clusters, but it does identify the clusters by color, and it allows the user to identify those clusters and name them, as I have done above. Further, you can explore your network by hovering over the individual dots that represent people.

Essentially, five groups arise from my network: Family, Friends, and Educators on the left; Dean Health Plan at the top; CPM Healthgrades at the upper right; the UW Health System, which is composed of the University of Wisconsin Medical Foundation (UWMF), the University of Wisconsin Hospitals and Clinics (UWHC), and the University of Wisconsin School of Medicine and Public Health (UWSMPH); and finally the SAS Institute at the bottom.

The lab isn't perfectly accurate, but it is pretty good. I checked out a number of individuals and some don't make sense, but most do. As an example, my wife is one of the larger dots on the left, which makes sense, since she also has networked with our friends and family on LinkedIn (although I usually avoid doing so for a number of reasons).

The UW Health cluster is visually split, but there is apparent movement and interrelationship between the organizations. There are some people who traveled between the UW and Dean, one way or another. The same is true between Dean and CPM, with most, I believe coming from Dean to CPM. There are some hubs in each organization, likely managers, project managers, or other people who attended a lot of meetings (I think one of the big dots at Dean was an IT manager who sat in on a lot of projects).

Aside from family and friends, SAS is probably the oddest group. I have some connections, and they seem to be somewhat strong. Before I attended the SAS Global Forum, this chart may have looked quite different, since I made many more connections after the conference. The chart also shows how well-developed my connections were at Dean and the UW, and how I'm still fairly new at CPM. (I bet if I ran this today it would look a bit better developed.)

Of course, charts like these leave out people who aren't on such sites as LinkedIn, but I would think that all the other people would compensate for them when graphed like this. Additionally, I don't have much of a network for old jobs like the ones I had in college, nor have I really networked much with fellow college classmates.

It would be interesting to see what other people's networks look like, especially people who are essentially professional networkers, like HR professionals or recruiters. How do networks in different industries look (mine is mostly health care)? What if you have someone who only networks with family or friends? What does that show? Perhaps different geographic locations you've lived in? How about someone who is a world traveler?

This type of graph is very powerful in that makes you think about the data behind it and ask such questions as I have done. It opens up doors we haven't thought of and inspires curiosity.

An Attempt to Answer a Quiz Question

Kyra heard the NPR Weekend Edition Sunday puzzle on the radio the other day: "Think of a common French word that everyone knows. Add a 'v' (as in 'violin') to the beginning and an 'e' at the end. The result will be the English-language equivalent of the French word. What is it?"

I thought, well, I don't know French, but I do know how to write a Python program that can look up the answer. Unfortunately, the puzzle appears to be a bit of a trick. I thought as much as I was writing the code. I was thinking, "I bet some people are going to be mad about the answer, because the question was worded in such a way as to be tricky."

At any rate, the code definitely does not result in the right answer, but it was a good exercise for me. First, I had to find two dictionaries robust enough to have a lot of words in English and French. Then I had to find dictionaries: I located the Debian operating system dictionary on my local computer, did a search in the package manager Synaptic for the French equivalent and any other English dictionaries that might help, and downloaded both. (The English alternative is labeled "insane" for its size, which "possibly contains invalid words (as well as words that are very uncommon)." [1])

Next I had to convert certain French characters into their English equivalent. To answer the question, I looked up all English words that start with "V" and end with "E", then removed those two letters, looking up the "word" in the modified French dictionary. It didn't work, as can be told from its output:

a
a
ah
aire
alu
an
ange
ares
as
er
erg
es
es
il
o
t

Yep, those are definitely not the answers. Some, when you add V and E back, aren't  even really English words. "Vte"? What is that? Oh, and I think A is listed twice due to the missing diacritics.

For anyone who's interested, I posted the code online. (It's not pretty posting a lot of code on Blogger, so here it is in a document.) Keep in mind I'm a new programmer, so I might not have done this task very elegantly. It is a short piece of code, though, and it could have worked.

One other note: If you attempt to run it, it may not work on your system. It's designed for Debian GNU/Linux, as it relies on dictionaries in a Debian system and it uses the "/" as the directory delimiter. Additionally, the "insane" dictionary makes it run a bit slow, and I did nothing in the code to help with that, like buffering the file, so you could run out of RAM or peak the CPU. It works on my system, which is all that mattered to me at the time.

Of course, a more robust program should be more careful and actually take user input in some fashion. Perhaps look up words that with other characters or use different languages. As it is, it's more of a "script" that does one thing and non-interactively. In any case, it doesn't even return the "right" answer!

[1] http://packages.debian.org/squeeze/wamerican-insane

Another Odd SAS Error Message

I encountered the following error message last week:

"ERROR: Invalid value for width specified - width out of range"

The code that generated it was using SQL and the PUT function to convert an ID using a format, like so:

compress(put(id, idfmt.), '. ')

Where the IDFMT format originated as a user-created format, stored in a data set. The quick and dirty solution is to add the "?" modifier to the PUT function, which is oddly documented on the INPUT function but not the PUT function:

compress(put(id, ? idfmt.), '. ')

Now why wouldn't that be documented on the PUT function page?

And what about my original problem? I've tried varying the lengths of the input column, the format, removing the compress statement, and switching to TRANWRD or PRXCHANGE. I also tried to identify the records involved, but with 35.6 million records, that was far to slow. I know that the issue occurred in the second half of the data, but beyond that is too time consuming to look for the needle in the haystack.

Finally, I figured out a solution: The original format is based on an input data set, using the CNTLIN= option on PROC FORMAT to generate the format on the fly. The data set does not have the HLO (High/Low/Other) flag to indicate what non-matching (Other) starting values would be labeled as. I added it like so:

data custom_format;

    /* Output the original records */
    set custom_format end=end;
    output;

    /* Output an additional record */
    if end then do;
        start=.;
        label=.;
        fmtname='idfmt';
        hlo='O';
        output;
    end;
run;

Then the code using the format worked, and to boot it no longer needed the COMPRESS function to remove the blanks and numeric missing values (".").

Hopefully anyone else with a similar issue will find that one of the above solutions fits their need.

SAS Enterprise Guide and the Log

I'm presenting on the CheckLog macro that I developed to review a log (or any file or list of files) for errors, warning, and other messages in SAS related to issues. To make sure everything is ready to go, I've been reviewing it to see if there's anything I can improve, especially in regard to SAS Enterprise Guide, the new(er) platform for using SAS.

I encountered a few issues between the two. First, the display manager statements (DM) no longer work in Enterprise Guide. The fix was easy, although a bit tricky: For Windows users (Enterprise Guide is currently only on Windows), I switched from using the DM POSTMESSAGE statement to a Windows API called MessageBox. The API allows greater flexibility than the POSTMESSAGE statement, e.g., I can set an icon to go along with the message. Now Windows users can tell what happened not only by the message, but the icon, too.

However, the DM statement that attempts to export the current log is still intact. I was unable to find a way to replace this, or much less export the Enterprise Guide log in a simple, programmatic fashion. Essentially, it is not possible to use CheckLog to review the "current log" as defined in the Base SAS sense. However, Enterprise Guide users can still check the "current log", but they have to make it an external log in one of two ways.

The first way is to export the log using PROC PRINTTO before the program executes. The code for this isn't too bad, and it can be easily added to any existing program:

/* Set a directory and filename for the log */

%let log=%sysfunc(pathname(work))\work.log;

/* Print the log to the specified location */

proc printto log="&LOG";

run;

/* Run some SAS code... */

proc sql;

create table test as

select *

from sashelp.vtable

;

quit;

/* Turn off printing to the file */

proc printto;

run;

/* Check the external log */

/* Note: The shadow is set to "N" (No) since it is not necessary to copy the log */

%checklog(&LOG, shadow=N);

This method works fairly well. However, not all users may want to use PROC PRINTTO, or they may be using it in a fashion that would conflict with the above suggestion.

The second option involves the project flow in Enterprise Guide. This solution is quite a bit more convoluted, and I outlined it, along with screenshots, on the CheckLog Enterprise Guide page. Essentially, the user has to export the log as a step in the project flow and run CheckLog in a small program. From there, the user can either programmatically determine what to do or set other steps in Enterprise Guide to run based on conditions, e.g., using the ISSUES macro variable to determine what to do next.

Overall, this is quite a bit more convoluted, and it would be great if there were some programmatic way to export the Enterprise Guide log. If anyone knows of a simple way to do that, please let me know!

SCAPROC Bug in SAS

In my current position, we have a macro set up to run other programs, and it outputs a copy of the code, an MPRINT file, and the log. It also re-runs a project-specific autoexec to keep a copy in the log, analyzes the log for errors and warnings (using something other than my CheckLog macro), and it collects and outputs metadata. The part that finds and outputs metadata is called PROC SCAPROC. Here is how SAS describes it:

The SCAPROC procedure implements the SAS Code Analyzer, which captures information about input, output, and the use of macro symbols from a SAS job while it is running. The SAS Code Analyzer can write this information and the information that is in the original SAS file to a file that you specify.

Sounds nifty, right? Well, I've noticed some odd issues with it over time, and only recently was I able to put my finger on one of its problems.

The arrangement of code that results in the issue is rather odd and easily avoided. Nonetheless, in case anyone has encountered the problem, I thought I would post code to replicate the problem here. First, run this part in a new SAS session:

/* Set the base directory */

%let dir=%sysfunc(pathname(work));

/* Create a new folder */

%sysexec mkdir "&DIR.\lock";

/* Assign a library with a locked data set */

libname lock "&DIR.\lock";

/* Create a locked data set */

data lock.locked(pw='asdf1234');

    x=1;

run;

So we've created a new library with a locked data set based on our work library. Now, try out SCAPROC with the following code and see what happens:

/* Copy the CARS data set */
data cars;
    set sashelp.cars;
run;

/* Create a view */
proc sql;
    create view badview as
    select type
    from cars
    ;
quit;

/* Remove type from the data set */
/* This makes the view (i)nvalid */
data cars;
    set cars;
    drop type;
run;

/* Start PROC SCAPROC */
proc scaproc;
    record "&DIR.\lock\scaproc_out.txt" attr;
run;

/* Copy data out of SASHELP */
data test;
    set sashelp.vcolumn(where=(libname="WORK"));
run;

/* Write out SCAPROC */
proc scaproc;
    write;
run;

And now we've created a view from a SASHELP table and made it invalid by removing the column that the view uses in its definition. We started SCAPROC and attempted to use another SASHELP table that contains metadata about all of the columns available to SAS. Did you get a pop-up for a password at this point? If so, that's part of the bug. Next, SCAPROC ends. Take a look at the log: are there some error messages? There should be at least one regarding the invalid view. Now take a look at the output of SCAPROC. It's a bit inflated for such a small program, especially if you have a large number of libraries assigned. Assign a bunch of libraries and try the code again, and it will explode.

The issue here has a few parts: First, we need an invalid view and a fully password-protected data set. Second, we need to run SCAPROC; and finally, we need to run a DATA step against SASHELP. The invalid view causes SCAPROC, upon encountering the SASHELP.Vcolumn data set, to consider all data sets listed in the table as input to the program. Interestingly, if you replace the DATA step with SQL, the issue disappears, except for the error regarding the invalid view. Dropping the use of SCAPROC also removes the problem (obviously).

Why did the password pop-up appear? Since SCAPROC considered all data sets as input, it attempted to collect metadata on the locked data set. A sub-bug of the main bug is that SCAPROC should not query locked data sets, since it won't know the password. If you were to run it in batch mode, there wouldn't be anyone around to enter a password. Another sub-bug is that the DATA step somehow tells SCAPROC that all available data is an input.

As a consequence of this issue, we had a program run where the metadata output of SCAPROC that contained variable names was a SAS data set with 5.1 million rows at 972 MB. That was huge, and the SCAPROC output file was also huge, at 751 MB. And it's just a text file - no SAS header, column structure, anything!

I submitted the bug set to SAS, but in the meantime if anyone encounters this issue, the easiest way to fix it is to either fix the broken views, replace the DATA step with SQL, or both. Good luck, and please let me know if you encounter the issue yourself, or any related issues!

News and Advice

Last week, I started a new position with the Health Innovation Program in the School of Medicine and Public Health at the University of Wisconsin-Madison. In this position, I hope to be more involved in research that could potentially affect patient care. Ever since my days doing academic and educational research, I have missed being involved in the process.

Along with this transition, it appears I've started to become a help desk of sorts for some SAS. I received two sets of email from different users of my macros, and I thought I would start sharing those questions and answers here:

Question 1:

Is there a way to limit the number of observations read by PROC IMPORT? I tried using OPTIONS OBS=2 before proc import and it worked but still the reduction in time was not significant.

Answer 1:

It depends on the file, so you'd have to look at the IMPORT procedure documentation depending on what you're importing to see if that feature is available.

Where did you put the option? It must have been just above the output. If so, then yes, it wouldn't decrease the time, since SAS is still pulling in all of the records. Depending on the location of the option, SAS may only keep the number of records on its way out of the import process. Either way, SAS may need to read the entire file, which would reduce time.

What if you tried just pulling in the header record, and delimiting them yourself? I'm guessing you're looking at comma-delimited, space-delimited, or tab-delimited data, and you should be able to figure out how to read the data in as one record, then split the variable names into multiple variables or even records. Then you could see if there are any new records.

Question 2:

Kindly help me to create the folders using SAS codes in local drive or on server without using X-command.

Answer 2:

You can use either CALL SYSTEM within a data step or %SYSEXEC. You'll want to tinker with the XSYNC and XWAIT options as you see fit. Here are two examples:

data _null_;

    call system('md c:\TestDir');

run;

OR

%sysexec(md c:\TestDir\);

What About Gas Prices?

Recently there's been a lot of hubbub about gas prices. While high, gas prices are not as high as many other things people buy, especially some things people buy on a daily basis, like coffee. For my family, I can demonstrate that the cost of gas is not as big a concern and not as big a budget item. Using Mint.com, my credit and debit cards, my wife's credit and debit cards, and many other financial accounts are linked and the transactions pooled together and categorized. Using this tool, I graphed the entire year of spending in 2010 in seconds:



Note: "Everything Else" includes Gifts and Donations, Health and Fitness, Uncategorized (cash and some checks), Travel, Business Services, Personal Care, Pets, Entertainment, and Fees and Charges. I also separated "Gas and Fuel" from "Auto and Transport" to demonstrate my point. What's left in in "Auto and Transport" includes Parking, Services and Parts, Auto Insurance, and "Other" (state transportation fees). Restaurants, coffee shops, and alcohol were included in Food and Dining, but I split those out into "Restaurants, etc.".

Our family probably does not spend like other families. We're generally fiscally conservative. We do like to buy good, healthy, and (when we can) local food, so our food prices are high. We don't splurge on electronics, entertainment, fancy phone or cable services, or personal care. We also minimize driving, which keeps auto costs lower. Both our drives to work are within 15 minutes. On the other hand we have a lot of educational debt. Even so, that's still a relatively small percentage of expenses. It should be no surprise that the top of the list is housing. That's probably true for just about anyone at any income level.

So what concerns me more? The cost of fuel or something else? Even as the cost of gas has risen, I'm not as concerned as I am about the cost of food, which is partly related to the cost of gas. However, there are more insidious factors in the cost of food: Wall Street.

As quoted by BoingBoing author Mark Frauenfelder, Frederick Kaufman wrote, in an article titled How Goldman Sachs Created the Food Crisis:

[T]he boom in new speculative opportunities in global grain, edible oil, and livestock markets has created a vicious cycle. The more the price of food commodities increases, the more money pours into the sector, and the higher prices rise. Indeed, from 2003 to 2008, the volume of index fund speculation increased by 1,900 percent. "What we are experiencing is a demand shock coming from a new category of participant in the commodities futures markets," hedge fund Michael Masters testified before Congress in the midst of the 2008 food crisis.

The result of Wall Street's venture into grain and feed and livestock has been a shock to the global food production and delivery system. Not only does the world's food supply have to contend with constricted supply and increased demand for real grain, but investment bankers have engineered an artificial upward pull on the price of grain futures. The result: Imaginary wheat dominates the price of real wheat, as speculators (traditionally one-fifth of the market) now outnumber bona-fide hedgers four-to-one.

Today, bankers and traders sit at the top of the food chain -- the carnivores of the system, devouring everyone and everything below. Near the bottom toils the farmer. For him, the rising price of grain should have been a windfall, but speculation has also created spikes in everything the farmer must buy to grow his grain -- from seed to fertilizer to diesel fuel. At the very bottom lies the consumer. The average American, who spends roughly 8 to 12 percent of her weekly paycheck on food, did not immediately feel the crunch of rising costs. But for the roughly 2-billion people across the world who spend more than 50 percent of their income on food, the effects have been staggering: 250 million people joined the ranks of the hungry in 2008, bringing the total of the world's "food insecure" to a peak of 1 billion -- a number never seen before.

As you can see in my spending graph, "Food and Dining" consumes 13% of our budget, higher than the quoted 8-12%. This category originally included restaurants, coffee shops, and alcohol, which I moved into "Restaurants, etc.", and was at 18.5% of spending. With these items removed, it seems much more reasonable, but still high. About 4 times higher than gas prices!

Given that our expenditure is already higher than the estimate in the article, and that food prices are likely to increase due to this investment scheme as well as increased transportation costs due to gas prices, I am much more concerned about the cost of food. Occasionally I joke that my son eats so much I might as well buy a farm to feed him, and that's not that far from the truth. We do partake in local farmer's markets and a community farm, but even so, those prices will probably be affected too.

So take a look at your own expenses and critically examine what's really costly. And once you figure that out, take action to reduce the costs however you can. For me, I'll chose to vote for politicians who push for financial reform and continue to buy as much local, organic food as I can.

Sound Advice for LinkedIn

The Job Doc has some good advice for how to behave on LinkedIn:

Avoid these 4 mistakes with LinkedIn

1) Never inviting anyone.

Hmm, this one's tempting. No one likes rejection; it forces you to relive high school. ... Last week, I invited 20 people to LinkedIn. Most accepted, a few ignored me. Nothing terrible happened.

2) Bad timing.

... Timing matters. Are your LinkedIn invitations too late, or too early? "Don't invite within two hours of meeting," says Rod Hughes, Director of Communications, Oxford Communications. "I typically wait till the next day," advises Rod. "Anything sooner seems stalker-esque."

3) Inviting everyone.

... "You need a policy," says Thom Singer, author of several networking books. "My policy," says Thom, "is The Coffee, Meal or Beer Rule, which means not accepting links unless I've had a real conversation."

4) Bad invitation.

At LinkedIn, the default invite is, "I'd like to add you to my professional network." But that's robotic. ... Tip: For better results, deliver a better invitation.

How do you go about inviting people into your network? Could you improve the way you go about it? I know I've used the default invitation far too often: usually because I'm in a hurry. Even a more personal default would be better than the LinkedIn default.

LinkedIn isn't Facebook either, where you can invite anyone on a whim. I've received a number of unwanted or unexpected invitations, some of which I probably should not have accepted given the "coffee, meal, or bear" rule. Those contacts I put in a "monitor" category, since I do not know these people very well and I'm not sure I want the association. Should their behavior be questionable, I remove the association with the contact.

What about you? How do you go about adding contacts?

Checking for Issues in SAS

I've been working on a macro program to check the SAS log for specific "issue" keywords, including "error", "warning", "invalid", and "uninitialized". Typically these are associated with issues in the program. I'm considering adding the following statements as well:

• "The query requires remerging summary statistics back with the original data"
• "Numeric values have been converted to character values"
• "Character values have been converted to numeric values"

The above statements may not always be associated with issues; however, there are better ways to write the program then to allow these statements to occur. For example, if the programmer wants to set a character variable to the text value of a numeric variable, it is better to use a put statement then to allow SAS to do it, as it may convert it incorrectly. Here's an example:

data _null_;
    today=today();
    call symput('today',today);
run;
%put NOTE: Today is &today;

The example sets today's date as a macro variable called today. However, the output is just a number. Depending on how the macro variable is used, this may be okay. Here's a better way to write it:

data _null_;
    today=today();
    call symput('today',compress(put(today,8.)));
run;
%put NOTE: The date value of today is &today;

Or, even simpler:

data _null_;
    today=today();
    call symputx('today',today);
run;
%put NOTE: The date value of today is &today;

Adding the "x" to the call symput statement will automatically set the numeric variable to character and remove spaces.

If the programmer actually intends to write out the date in a date format, then the following code will work. (Note that the date format can be whatever format desired. I like the YYMMDDN8 format since it sorts better.)

data _null_;
    today=today();
    call symput('today',put(today,yymmddn8.));
run;
%put NOTE: Today is &today;

The macro program I've written can identify these types of issues so they can be corrected. I've also recently discovered that other SAS users have created such log checking utilities, and they include more phrases that would indicate issues within various procedures that I do not use frequently. Once I have time to review these statements, I will include them in my program as well.

In addition to identifying these issues, I'm filtering out common issues that occur with the above keywords that are probably not associated with issues in the program. For example, a libname error from an autoexec program may not be an error for the program if it is not used. If it is used, there will be subsequent errors that would identify the issue.

This type of non-issue occurs at work frequently. We have a generic autoexec that runs when SAS is started, and it includes libnames that are located in directories I do not have the security rights to view. Therefore, every time I start SAS there are errors in the log. When I check the log for issues, I do not want these generic issues to appear.

How do you check for issues in a SAS program? Do you have a utility for doing so? What do you check for?

Internet Explorer 6 at Work

At my last and current job, my computers have been set up with Internet Explorer 6 as the default browser, even though Microsoft has developed Internet Explorer 7 and version 8, not to mention the flourish of alternative browsers, like Firefox, Safari, and Google Chrome, to mention only a few.

I've noticed that the possible reason for this may be legacy software, software that may exploit some security hole in Internet Explorer 6 in order to operate. At my last job, I downloaded Internet Explorer 7 because I thought it would be more secure while I was performing research. (At my new job, the IT department is much more serious about security, since the company deals with health care data. I am unable to modify much of anything. My screen resolution isn't even permanent!)

When I tried to access one of the company's intranet pages for storing customer information, I received numerous errors while trying to use some of the features. I tried lowering the security settings, updating all sorts of related software, and modifying the connection settings, all to no avail. It just wasn't going to work on Internet Explorer 7.

The software has some serious flaws - it's number 8 on the worst tech products of all time rated by PCWorld and the Computer Emergency Response Team has recommended that internet users not use it at all. So I'm wondering, with all of the security flaws and out-of-date features, why do companies still use it? Do newer versions lack compatibility with legacy software? Is it too difficult and time-consuming to convert to the newest version? Is it easier just to wait until Vista is the defacto operating system for business PCs? What is the reason?

Update: One of my co-workers had IE 8 installed. It was an attempt to help a web interface work better, and it seemed to do the trick! Lesson: Upgrade!